Oct 31 03:06:16.383084 ip-10-0-134-83 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 03:06:16.830085 ip-10-0-134-83 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:06:16.830085 ip-10-0-134-83 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 03:06:16.830085 ip-10-0-134-83 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:06:16.830085 ip-10-0-134-83 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:06:16.833541 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833451    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:16.833541 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833537    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:16.833541 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833542    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:16.833541 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833546    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:16.833541 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833551    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833556    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833560    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833564    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833567    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833571    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833575    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833579    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833583    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833587    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833597    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833601    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833605    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833610    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833615    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833618    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833622    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833626    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833630    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833633    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:16.833852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833637    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833641    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833645    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833649    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833653    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833657    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833661    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833665    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833669    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833673    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833678    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833682    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833686    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833691    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833696    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833702    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833706    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833710    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833714    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833719    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:16.834663 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833723    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833727    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833731    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833736    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833740    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833744    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833748    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833753    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833757    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833761    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833766    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833773    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833778    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833783    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833803    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833807    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833812    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833817    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833821    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:16.835512 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833825    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833829    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833834    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833838    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833842    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833846    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833853    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833860    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833864    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833870    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833875    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833880    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833885    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833889    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833893    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833898    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833902    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833907    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833911    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:16.836327 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833916    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833920    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833926    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833931    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833935    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833939    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833944    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833948    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833953    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833957    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833961    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833966    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833970    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833974    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833978    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833982    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833986    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833991    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.833995    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:16.837154 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.834000    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.834006    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.834011    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.834015    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.834021    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835295    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835305    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835310    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835315    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835320    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835324    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835329    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835333    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835337    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835342    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835346    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835350    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835355    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835360    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835364    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:16.837973 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835368    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835372    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835377    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835381    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835385    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835389    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835393    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835402    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835407    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835411    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835416    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835420    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835425    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835429    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835434    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835441    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835447    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835454    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835459    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:16.838895 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835463    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835468    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835472    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835476    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835481    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835486    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835490    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835495    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835500    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835504    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835508    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835511    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835515    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835519    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835524    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835528    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835532    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835537    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835542    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835546    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:16.839698 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835552    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835556    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835561    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835565    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835569    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835574    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835578    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835583    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835587    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835591    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835596    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835601    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835605    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835609    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835614    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835618    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835622    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835626    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835630    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:16.840281 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835635    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835639    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835643    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835647    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835652    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835656    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835660    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835664    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835668    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835672    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835676    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835681    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835685    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835690    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835694    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835698    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835702    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835706    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835712    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:16.840930 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835716    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835721    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835725    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835729    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835741    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835746    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835761    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835767    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835772    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835776    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835781    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835802    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835806    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.835810    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836641    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836649    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836654    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836659    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836663    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:16.841389 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836668    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836672    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836676    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836681    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836686    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836690    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836694    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836699    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836703    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836707    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836711    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836716    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836721    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836725    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836730    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836734    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836738    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836742    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836746    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836751    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:16.841953 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836756    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836760    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836765    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836769    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836773    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836777    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836781    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836802    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836807    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836811    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836815    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836820    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836824    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836828    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836832    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836836    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836841    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836846    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836850    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:16.842421 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836855    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836859    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836863    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836867    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836872    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836876    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836881    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836885    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836889    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836893    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836898    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836904    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836910    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836918    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836924    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836928    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836933    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836937    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836941    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:16.842876 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836945    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836949    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836954    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836958    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836962    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836966    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836970    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836974    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836979    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836983    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836987    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836991    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.836995    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837000    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837005    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837009    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837013    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837020    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837024    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837029    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:16.843437 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837033    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837038    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837043    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837047    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837052    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837056    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837060    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837065    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837069    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837073    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837078    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837082    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837086    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837090    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837095    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837099    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837104    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837108    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837112    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:16.844029 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837116    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837124    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837129    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.837134    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837271    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837283    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837293    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837301    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837308    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837313    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837323    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837330    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837335    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837342    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837347    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837353    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837358    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837363    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837367    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837372    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837377    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837383    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837389    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 31 03:06:16.844467 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837394    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837399    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837404    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837409    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837415    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837420    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837427    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837433    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837438    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837443    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837448    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837453    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837457    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837465    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837470    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837475    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837480    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837485    2577 flags.go:64] FLAG: --enable-server="true"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837490    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837500    2577 flags.go:64] FLAG: --event-burst="100"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837505    2577 flags.go:64] FLAG: --event-qps="50"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837510    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837516    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837521    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 31 03:06:16.845026 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837528    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837532    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837537    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837542    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837547    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837552    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837557    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837562    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837567    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837572    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837577    2577 flags.go:64] FLAG: --feature-gates=""
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837584    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837589    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837594    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837604    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837609    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837614    2577 flags.go:64] FLAG: --help="false"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837619    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837624    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837630    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837634    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837640    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837646    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 03:06:16.845661 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837650    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837655    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837660    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837665    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837671    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837677    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837681    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837686    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837691    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837696    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837700    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837705    2577 flags.go:64] FLAG: --lock-file=""
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837710    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837715    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837721    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837730    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837735    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837740    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837745    2577 flags.go:64] FLAG: --logging-format="text"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837750    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837755    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837760    2577 flags.go:64] FLAG: --manifest-url=""
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837765    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837774    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837779    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 03:06:16.846228 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837800    2577 flags.go:64] FLAG: --max-pods="110"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837806    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837811    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837815    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837820    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837825    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837830    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837835    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837845    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837850    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837855    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837864    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837869    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837873    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837878    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837883    2577 flags.go:64] FLAG: --port="10250"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837888    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837892    2577 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-06d809d645533215d"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837898    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837903    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837908    2577 flags.go:64] FLAG: --register-node="true"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837913    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837919    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837924    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837928    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837934    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 31 03:06:16.846826 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837940    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837945    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837950    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837955    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837961    2577 flags.go:64] FLAG: --runonce="false"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837966    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837971    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837975    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837980    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837984    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837989    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837994    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.837999    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838004    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838009    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838014    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838018    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838025    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838030    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838034    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838043    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838048    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838053    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838061    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838066    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 03:06:16.847466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838071    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838076    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838081    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838086    2577 flags.go:64] FLAG: --v="2"
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838093    2577 flags.go:64] FLAG: --version="false"
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838100    2577 flags.go:64] FLAG: --vmodule=""
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838106    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838113    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838333    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838340    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838345    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838352    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838357    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838361    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838365    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838369    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838374    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838379    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838383    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838388    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838392    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:16.848094 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838396    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838401    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838406    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838412    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838418    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838422    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838426    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838430    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838437    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838442    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838447    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838451    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838455    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838460    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838464    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838468    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838473    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838477    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838481    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:16.848597 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838486    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838491    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838496    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838501    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838506    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838510    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838514    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838518    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838523    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838527    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838531    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838535    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838540    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838544    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838548    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838553    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838559    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838564    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838568    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838572    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:16.849105 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838577    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838581    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838585    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838589    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838594    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838598    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838602    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838606    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838611    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838615    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838620    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838624    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838630    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838639    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838645    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838652    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838656    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838661    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838665    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:16.849590 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838669    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838674    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838678    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838682    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838686    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838691    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838695    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838699    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838703    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838709    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838713    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838718    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838723    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838727    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838731    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838735    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838740    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838744    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838748    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838752    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:16.850064 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838756    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838760    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838765    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838769    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838773    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838777    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838782    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838802    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838809    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838813    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838818    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838822    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838826    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838830    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838834    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.838880    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:06:16.850531 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838941    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838964    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838970    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838974    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838978    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838983    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838990    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838994    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.838998    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839002    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839007    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839011    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839015    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839019    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839023    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839027    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839032    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839037    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839041    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839045    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:16.850941 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839050    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839054    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839059    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839063    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839075    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839082    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839089    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839094    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839098    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839103    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839107    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839111    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839115    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839120    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839124    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839128    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839133    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839137    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839143    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:16.851409 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839147    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839151    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839156    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839160    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839164    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839169    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839173    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839177    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839181    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839185    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839190    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839194    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839198    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839203    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839207    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839211    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839215    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839221    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839233    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839238    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:16.851890 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839242    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839246    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839250    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839255    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839259    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839263    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839268    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839272    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839276    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839280    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839284    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839291    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839310    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839314    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839319    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839323    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839327    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839332    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839336    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839341    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:16.852363 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839345    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839349    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839353    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839358    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839363    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839367    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839371    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839375    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839379    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839388    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839395    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839404    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839409    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839414    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839419    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839424    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839428    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839432    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839438    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:16.852841 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839442    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839447    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839451    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839455    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839461    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839465    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839470    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.839474    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.847216    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.847232    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.847247    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.847252    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847311    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847317    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847320    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847322    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847325    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847328    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847331    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847333    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:16.853324 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847336    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847339    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847342    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847344    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847347    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847350    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847352    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847355    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847358    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847360    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847363    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847366    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847368    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847371    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847373    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847376    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847379    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847381    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847384    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:16.853819 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847387    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847390    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847399    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847403    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847405    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847408    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847411    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847414    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847417    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847419    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847422    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847424    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847427    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847429    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847432    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847435    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847437    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847440    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847442    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:16.854378 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847445    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847447    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847450    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847452    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847456    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847466    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847469    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847472    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847475    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847478    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847481    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847484    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847486    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847489    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847492    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847495    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847504    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847507    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847510    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:16.854997 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847513    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847515    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847518    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847520    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847523    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847525    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847528    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847531    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847533    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847536    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847539    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847541    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847544    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847546    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847549    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847551    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847554    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847556    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847559    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847561    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:16.855691 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847564    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847567    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847569    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847572    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847574    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847577    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847579    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847582    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847587    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847591    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847600    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847603    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847606    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847608    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847611    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847614    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847616    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847619    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847621    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847624    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:16.856255 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847626    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.847654    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847871    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847877    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847880    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847884    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847886    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847889    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847892    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847894    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847897    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847899    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847902    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847905    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847907    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847910    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:16.856871 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847913    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847915    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847918    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847920    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847924    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847927    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847929    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847938    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847941    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847944    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847947    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847949    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847952    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847954    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847957    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847960    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847962    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847965    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847968    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:16.857568 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847970    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847973    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847975    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847978    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847980    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847983    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847987    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847990    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847993    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847997    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.847999    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848002    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848005    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848007    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848010    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848014    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848017    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848021    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:16.858342 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848023    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848026    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848029    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848037    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848040    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848042    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848045    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848047    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848050    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848052    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848055    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848057    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848060    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848063    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848065    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848068    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848070    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848072    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848075    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848078    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:16.859118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848080    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848083    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848085    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848088    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848091    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848093    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848096    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848099    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848101    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848104    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848107    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848110    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848112    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848115    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848118    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848121    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848129    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848132    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848134    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:16.859888 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848137    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848139    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848142    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848144    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848147    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848150    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848152    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848155    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848157    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848159    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848162    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848164    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848167    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848169    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848172    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848175    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:16.860332 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.848199    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848237    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848241    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848244    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848247    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848249    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848252    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848257    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848260    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848263    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848265    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848268    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848270    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848273    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848282    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848285    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848287    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848290    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848292    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:06:16.860803 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848295    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848298    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848300    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848303    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848305    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848308    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848311    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848313    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848316    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848318    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848320    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848323    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848325    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848328    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848330    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848333    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848336    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848338    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848341    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848343    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:06:16.861272 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848348    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848351    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848354    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848356    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848359    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848361    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848364    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848366    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848375    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848378    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848380    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848383    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848385    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848388    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848391    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848394    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848396    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848399    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848401    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848404    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:06:16.861954 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848406    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848409    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848412    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848421    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848424    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848426    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848429    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848432    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848434    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848437    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848440    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848442    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848447    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848449    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848452    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848455    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848457    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848460    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848462    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:06:16.862479 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848465    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848468    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848477    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848481    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848484    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848487    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848489    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848492    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848494    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848497    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848500    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848505    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848508    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848511    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848514    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848516    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848519    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848522    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848524    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:06:16.862958 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848527    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848529    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848532    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848535    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848538    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848541    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848546    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848549    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848551    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:16.848554    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.848617    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.850870    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.851807    2577 server.go:1014] "Starting client certificate rotation"
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.851932    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.851966    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.857934    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:06:16.863456 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.859703    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:06:16.871726 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.871707    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 31 03:06:16.875127 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.875111    2577 log.go:25] "Validated CRI v1 image API"
Oct 31 03:06:16.875982 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.875952    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 03:06:16.879108 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.879088    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b81b264a-72e5-490a-836a-1722ad0dc5bc:/dev/nvme0n1p4 d0dd9871-4a01-48fe-ad50-170d5903f4c7:/dev/nvme0n1p3]
Oct 31 03:06:16.879159 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.879108    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 03:06:16.883377 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.883260    2577 manager.go:217] Machine: {Timestamp:2026-10-31 03:06:16.882771256 +0000 UTC m=+0.369810913 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100407 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23dad2cbb57f637e48b7db7ee22a9e SystemUUID:ec23dad2-cbb5-7f63-7e48-b7db7ee22a9e BootID:c28f07c8-2d0d-48f2-809f-ddb4e5802190 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:c4:39:99:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:c4:39:99:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:c8:12:6c:7e:47 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 03:06:16.883377 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.883376    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 03:06:16.883499 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.883453    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 03:06:16.888630 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.888600    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 03:06:16.888796 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.888632    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-83.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 03:06:16.888850 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.888797    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 03:06:16.888850 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.888807    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 03:06:16.888850 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.888834    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:06:16.892333 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.892320    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:06:16.893369 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.893358    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 03:06:16.898127 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.898113    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 03:06:16.898249 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.898240    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 03:06:16.898947 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.898937    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 03:06:16.899046 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.898952    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 03:06:16.899046 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.898966    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 03:06:16.899046 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.898979    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 31 03:06:16.899046 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.898993    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 03:06:16.899563 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.899545    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:06:16.905025 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.904985    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 03:06:16.910979 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.910966    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:06:16.911903 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.911888    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 03:06:16.911967 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.911920    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 03:06:16.912082 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.912070    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 03:06:16.912125 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.912087    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 03:06:16.912125 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.912093    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 03:06:16.912125 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.912099    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 03:06:16.912125 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.912105    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 03:06:16.912125 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.912111    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 03:06:16.913074 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.913065    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 03:06:16.913074 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.913074    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 03:06:16.913129 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.913082    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 03:06:16.913129 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.913088    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 03:06:16.913129 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.913096    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 03:06:16.913129 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.913105    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 03:06:16.914589 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.914580    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 03:06:16.914589 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.914589    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 03:06:16.918369 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.918355    2577 server.go:1263] "Started kubelet"
Oct 31 03:06:16.918470 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.918439    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 03:06:16.918519 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.918442    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 03:06:16.918568 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.918520    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 03:06:16.919208 ip-10-0-134-83 systemd[1]: Started Kubernetes Kubelet.
Oct 31 03:06:16.919973 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.919958    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 03:06:16.921829 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.921814    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 03:06:16.923042 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:16.923020    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 03:06:16.926170 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.926157    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 03:06:16.926227 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.926180    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 03:06:16.926382 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:16.926363    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:06:16.926888 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.926778    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 03:06:16.926888 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.926808    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 03:06:16.926888 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.926883    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 03:06:16.927059 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.926952    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 03:06:16.927059 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.926958    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 03:06:16.927189 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.927172    2577 factory.go:55] Registering systemd factory
Oct 31 03:06:16.927240 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.927192    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 31 03:06:16.927435 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.927415    2577 factory.go:153] Registering CRI-O factory
Oct 31 03:06:16.927435 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.927435    2577 factory.go:223] Registration of the crio container factory successfully
Oct 31 03:06:16.927565 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.927484    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 03:06:16.927565 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.927506    2577 factory.go:103] Registering Raw factory
Oct 31 03:06:16.927565 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.927527    2577 manager.go:1199] Started watching for new ooms in manager
Oct 31 03:06:16.928498 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.928483    2577 manager.go:319] Starting recovery of all containers
Oct 31 03:06:16.939959 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.939939    2577 manager.go:324] Recovery completed
Oct 31 03:06:16.945210 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.945197    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:16.947197 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.947171    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:16.947267 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.947208    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:16.947267 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.947231    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:16.947910 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.947780    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 31 03:06:16.947980 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.947915    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 03:06:16.947980 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.947936    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 03:06:16.947980 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.947809    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:16.949442 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.949423    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:16.949517 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.949454    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:16.949517 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.949465    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:16.950189 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.950168    2577 policy_none.go:50] "Start"
Oct 31 03:06:16.950251 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.950196    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 03:06:16.950251 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.950212    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 03:06:16.952455 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.952442    2577 policy_none.go:44] "Start"
Oct 31 03:06:16.975438 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.975405    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 03:06:16.980049 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.980032    2577 manager.go:341] "Starting Device Plugin manager"
Oct 31 03:06:16.980146 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:16.980074    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 03:06:16.980146 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.980087    2577 server.go:87] "Starting device plugin registration server"
Oct 31 03:06:16.980360 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.980349    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 03:06:16.980392 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.980364    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 03:06:16.980524 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.980499    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 03:06:16.980631 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.980616    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 03:06:16.980631 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:16.980630    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 03:06:16.981202 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:16.981183    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 03:06:16.981302 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:16.981231    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:06:17.086747 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.048731    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-83.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 03:06:17.086747 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.048761    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-83.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 03:06:17.086747 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.048716    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-83.us-west-2.compute.internal.18e37e543eba252d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-83.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-83.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 03:06:16.918328621 +0000 UTC m=+0.405368277,LastTimestamp:2026-10-31 03:06:16.918328621 +0000 UTC m=+0.405368277,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-83.us-west-2.compute.internal,}"
Oct 31 03:06:17.086747 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.061182    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 03:06:17.086747 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.062404    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 03:06:17.086747 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.062424    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 03:06:17.086747 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.062441    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 03:06:17.086747 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.062501    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 03:06:17.086747 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.081125    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:17.086747 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.082518    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:17.086747 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.082552    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:17.086747 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.082568    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:17.086747 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.082593    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.100763 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.100721    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 03:06:17.100910 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.100817    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-83.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 03:06:17.100910 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.100734    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-83.us-west-2.compute.internal.18e37e5440729e8c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-83.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-83.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-83.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 03:06:16.947195532 +0000 UTC m=+0.434235188,LastTimestamp:2026-10-31 03:06:16.947195532 +0000 UTC m=+0.434235188,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-83.us-west-2.compute.internal,}"
Oct 31 03:06:17.109243 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.109220    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 03:06:17.109346 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.109262    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4nrfd"
Oct 31 03:06:17.163572 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.163529    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"]
Oct 31 03:06:17.163704 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.163609    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:17.164636 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.164619    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:17.164727 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.164648    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:17.164727 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.164685    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:17.167034 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.167019    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:17.167185 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.167171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.167218 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.167202    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:17.168405 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.168389    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:17.168458 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.168422    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:17.168458 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.168433    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:17.168518 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.168391    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:17.168518 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.168495    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:17.168518 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.168505    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:17.170540 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.170521    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.170610 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.170555    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:17.171378 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.171363    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:17.171444 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.171395    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:17.171444 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.171407    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:17.196042 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.196017    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:17.197274 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.197254    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:17.199798 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.199763    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found" node="ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.202242 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.202225    2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found" node="ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.227919 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.227888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/584715508ebcb798df8a2f82469ae3b6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"584715508ebcb798df8a2f82469ae3b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.242093 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.242071    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.242165 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.242096    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-83.us-west-2.compute.internal\": node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:06:17.328059 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.328024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d194eddb65e9ecc3707084f3a0166db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"9d194eddb65e9ecc3707084f3a0166db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.328211 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.328096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d194eddb65e9ecc3707084f3a0166db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"9d194eddb65e9ecc3707084f3a0166db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.328211 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.328151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/584715508ebcb798df8a2f82469ae3b6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"584715508ebcb798df8a2f82469ae3b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.328297 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.328201    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/584715508ebcb798df8a2f82469ae3b6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"584715508ebcb798df8a2f82469ae3b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.350388 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.350333    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:06:17.428836 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.428783    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d194eddb65e9ecc3707084f3a0166db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"9d194eddb65e9ecc3707084f3a0166db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.428836 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.428815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d194eddb65e9ecc3707084f3a0166db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"9d194eddb65e9ecc3707084f3a0166db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.429022 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.428859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d194eddb65e9ecc3707084f3a0166db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"9d194eddb65e9ecc3707084f3a0166db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.429022 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.428888    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d194eddb65e9ecc3707084f3a0166db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"9d194eddb65e9ecc3707084f3a0166db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.450942 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.450914    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:06:17.504262 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.504234    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.506374 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.506355    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:17.551528 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.551361    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:06:17.652444 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.652358    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:06:17.753108 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.753064    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:06:17.851447 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.851415    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 03:06:17.852010 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.851613    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 03:06:17.852010 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:17.851611    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 03:06:17.852010 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.851589    2577 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a9b46fe1a16de407895db29e9b878039-acea73c2d7879f85.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-134-83.us-west-2.compute.internal.18e37e544073431d\": read tcp 10.0.134.83:40598->18.189.77.5:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-134-83.us-west-2.compute.internal.18e37e544073431d  default   5151 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-83.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-83.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-83.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 03:06:16 +0000 UTC,LastTimestamp:2026-10-31 03:06:17.164691571 +0000 UTC m=+0.651731233,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-83.us-west-2.compute.internal,}"
Oct 31 03:06:17.853745 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.853727    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:06:17.954883 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:17.954810    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:06:18.055532 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:18.055507    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:06:18.113784 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.113748    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:06:18.114257 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.114217    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:01:17 +0000 UTC" deadline="2028-05-22 05:29:26.992834242 +0000 UTC"
Oct 31 03:06:18.114310 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.114257    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13658h23m8.87858029s"
Oct 31 03:06:18.114843 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.114827    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:06:18.114906 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.114859    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:06:18.114906 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.114870    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:06:18.156899 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:18.156861    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:06:18.257720 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:18.257627    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:06:18.309859 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.309820    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:18.311306 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.311283    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:18.327588 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.327561    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:18.388561 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.388526    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:06:18.389788 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.389773    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:06:18.447980 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.447948    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:06:18.875194 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.875160    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:18.902650 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.902613    2577 apiserver.go:52] "Watching apiserver"
Oct 31 03:06:18.927976 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.927951    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 03:06:18.938852 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:18.938812    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584715508ebcb798df8a2f82469ae3b6.slice/crio-8eeb180f70ec034f4e3eea020d257929f75b83377a99352b01db8234a02d3ca7 WatchSource:0}: Error finding container 8eeb180f70ec034f4e3eea020d257929f75b83377a99352b01db8234a02d3ca7: Status 404 returned error can't find the container with id 8eeb180f70ec034f4e3eea020d257929f75b83377a99352b01db8234a02d3ca7
Oct 31 03:06:18.946056 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:18.946036    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:06:18.951505 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:18.951481    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d194eddb65e9ecc3707084f3a0166db.slice/crio-45f57db50ba1789cd6bb63193217cd822742e83154a9cfeb674d13ce7e0b0056 WatchSource:0}: Error finding container 45f57db50ba1789cd6bb63193217cd822742e83154a9cfeb674d13ce7e0b0056: Status 404 returned error can't find the container with id 45f57db50ba1789cd6bb63193217cd822742e83154a9cfeb674d13ce7e0b0056
Oct 31 03:06:19.007807 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.007775    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 03:06:19.008124 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.008106    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-28rvl","openshift-image-registry/node-ca-sdwgq","openshift-multus/multus-6vx62","openshift-network-diagnostics/network-check-target-mmv7s","kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-xm5zs","kube-system/global-pull-secret-syncer-ndfgt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh","openshift-cluster-node-tuning-operator/tuned-fwmjk","openshift-multus/multus-additional-cni-plugins-brs5l","kube-system/konnectivity-agent-c7b69","openshift-dns/node-resolver-4fpkl","openshift-multus/network-metrics-daemon-58h48"]
Oct 31 03:06:19.011972 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.011952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:06:19.014278 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.014261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:06:19.016578 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.016557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.018875 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.018851    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:06:19.021117 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.021098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:06:19.023463 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.023442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.025985 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.025969    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.028442 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.028423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:19.028542 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.028499    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:19.030953 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.030937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.033588 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.033566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.036034 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.036015    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:19.036130 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.036077    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:19.037263 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037238    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5c68fab-89c6-485c-a250-cb958115e5a2-konnectivity-ca\") pod \"konnectivity-agent-c7b69\" (UID: \"f5c68fab-89c6-485c-a250-cb958115e5a2\") " pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:06:19.037346 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.037346 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhdgp\" (UniqueName: \"kubernetes.io/projected/2cb0ab42-11a8-493c-8218-4755fb32e543-kube-api-access-rhdgp\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.037440 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-systemd-units\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.037440 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f714c9b3-ccf2-4dab-9220-321c5b4258d0-kubelet-config\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:19.037440 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037416    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-cni-bin\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-registration-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-run-ovn-kubernetes\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-env-overrides\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037613    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-netns\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-etc-kubernetes\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6k56\" (UniqueName: \"kubernetes.io/projected/a957db89-5022-4877-aa05-3713401881ed-kube-api-access-l6k56\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-cnibin\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-hostroot\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037913    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f714c9b3-ccf2-4dab-9220-321c5b4258d0-dbus\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-cni-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-conf-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.037986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-daemon-config\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.038344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-run-netns\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-cni-netd\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-socket-dir-parent\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-device-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-sys-fs\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-ovn\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038286    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-node-log\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/90b722fb-cc4e-4c62-877b-db0df5c5d16c-cni-binary-copy\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056de880-e0a6-4b20-805f-669a017b3d1d-tmp-dir\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-socket-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-kubelet\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-systemd\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-log-socket\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-kubelet\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-host\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:06:19.039054 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovnkube-config\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.038551    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038561    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovnkube-script-lib\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-cni-multus\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5c68fab-89c6-485c-a250-cb958115e5a2-agent-certs\") pod \"konnectivity-agent-c7b69\" (UID: \"f5c68fab-89c6-485c-a250-cb958115e5a2\") " pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovn-node-metrics-cert\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcl8m\" (UniqueName: \"kubernetes.io/projected/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-kube-api-access-zcl8m\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038659    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-multus-certs\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038674    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/056de880-e0a6-4b20-805f-669a017b3d1d-hosts-file\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038688    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-slash\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbbg\" (UniqueName: \"kubernetes.io/projected/90b722fb-cc4e-4c62-877b-db0df5c5d16c-kube-api-access-xlbbg\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-serviceca\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dbdb\" (UniqueName: \"kubernetes.io/projected/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-kube-api-access-7dbdb\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038810    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-cni-bin\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a957db89-5022-4877-aa05-3713401881ed-host-slash\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-system-cni-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.039545 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-etc-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.040057 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-k8s-cni-cncf-io\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.040057 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-var-lib-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.040057 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a957db89-5022-4877-aa05-3713401881ed-iptables-alerter-script\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:06:19.040057 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.038998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shnrb\" (UniqueName: \"kubernetes.io/projected/056de880-e0a6-4b20-805f-669a017b3d1d-kube-api-access-shnrb\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:06:19.064918 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.064808    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:06:19.064918 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.064845    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 03:06:19.065093 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.064941    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9g6gp\""
Oct 31 03:06:19.065093 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.064972    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 03:06:19.066212 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.066119    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 03:06:19.066212 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.066134    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 03:06:19.066212 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.066167    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lsfgs\""
Oct 31 03:06:19.067742 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.067696    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal" event={"ID":"9d194eddb65e9ecc3707084f3a0166db","Type":"ContainerStarted","Data":"45f57db50ba1789cd6bb63193217cd822742e83154a9cfeb674d13ce7e0b0056"}
Oct 31 03:06:19.067915 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.067899    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 03:06:19.068263 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.068242    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 03:06:19.068391 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.068273    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 03:06:19.068391 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.068292    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8l9rs\""
Oct 31 03:06:19.069351 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.069332    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal" event={"ID":"584715508ebcb798df8a2f82469ae3b6","Type":"ContainerStarted","Data":"8eeb180f70ec034f4e3eea020d257929f75b83377a99352b01db8234a02d3ca7"}
Oct 31 03:06:19.104790 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.104759    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 03:06:19.105074 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.104894    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 03:06:19.105074 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.104933    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 03:06:19.105074 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.104985    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 03:06:19.105074 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.105002    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 03:06:19.105321 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.105123    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 03:06:19.105321 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.105133    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 03:06:19.105321 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.105127    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 03:06:19.105321 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.105164    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bdmwz\""
Oct 31 03:06:19.105321 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.105164    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 03:06:19.105321 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.105210    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 03:06:19.105321 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.105261    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2wckk\""
Oct 31 03:06:19.105321 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.105310    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gtdck\""
Oct 31 03:06:19.108477 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.108464    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 03:06:19.108620 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.108606    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 03:06:19.108666 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.108650    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 03:06:19.108764 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.108752    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 03:06:19.114764 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.114730    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-95c6m\""
Oct 31 03:06:19.114764 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.114754    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 03:06:19.114764 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.114757    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:06:19.114971 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.114828    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l8z7f\""
Oct 31 03:06:19.114971 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.114847    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 03:06:19.114971 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.114848    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 03:06:19.114971 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.114872    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 03:06:19.114971 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.114945    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 03:06:19.115138 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.114995    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-47gs9\""
Oct 31 03:06:19.115138 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.115007    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rz4nz"
Oct 31 03:06:19.127636 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.127610    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 03:06:19.140150 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140117    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-systemd-units\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.140354 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f714c9b3-ccf2-4dab-9220-321c5b4258d0-kubelet-config\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:19.140354 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-cni-bin\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.140354 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.140354 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-registration-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.140354 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-run-ovn-kubernetes\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.140354 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140256    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-systemd-units\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.140354 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140284    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-cni-bin\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.140354 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-env-overrides\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.140354 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140288    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.140354 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-netns\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.140354 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140347    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-netns\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.140354 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140288    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f714c9b3-ccf2-4dab-9220-321c5b4258d0-kubelet-config\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-etc-kubernetes\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-lib-modules\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-var-lib-kubelet\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-system-cni-dir\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140450    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-etc-kubernetes\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6k56\" (UniqueName: \"kubernetes.io/projected/a957db89-5022-4877-aa05-3713401881ed-kube-api-access-l6k56\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-cnibin\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140636    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-cnibin\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-hostroot\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-host\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140692    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-hostroot\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwwgj\" (UniqueName: \"kubernetes.io/projected/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-kube-api-access-xwwgj\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140860    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-run-ovn-kubernetes\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140870    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-registration-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140862    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-env-overrides\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.140909 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140886    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f714c9b3-ccf2-4dab-9220-321c5b4258d0-dbus\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-cni-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-conf-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-daemon-config\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.140999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-systemd\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141006    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-conf-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-run-netns\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141028    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-cni-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141048    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f714c9b3-ccf2-4dab-9220-321c5b4258d0-dbus\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141060    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-run-netns\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-cni-netd\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141084    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141085    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-cni-netd\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysctl-conf\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.141546 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-socket-dir-parent\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.141234    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-device-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-sys-fs\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.141301    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret podName:f714c9b3-ccf2-4dab-9220-321c5b4258d0 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:19.641279697 +0000 UTC m=+3.128319361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret") pod "global-pull-secret-syncer-ndfgt" (UID: "f714c9b3-ccf2-4dab-9220-321c5b4258d0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141310    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-sys-fs\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141320    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-run\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141332    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-device-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141329    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-socket-dir-parent\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-ovn\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-node-log\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/90b722fb-cc4e-4c62-877b-db0df5c5d16c-cni-binary-copy\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141419    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-node-log\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141441    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-ovn\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056de880-e0a6-4b20-805f-669a017b3d1d-tmp-dir\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-socket-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.142342 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-kubelet\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141518    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-kubernetes\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141540    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-kubelet\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-systemd\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-log-socket\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141574    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-systemd\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-kubelet\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141593    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-log-socket\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysctl-d\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141618    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-kubelet\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-host\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141634    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-host\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovnkube-config\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovnkube-script-lib\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141717    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-cni-multus\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141751    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5c68fab-89c6-485c-a250-cb958115e5a2-agent-certs\") pod \"konnectivity-agent-c7b69\" (UID: \"f5c68fab-89c6-485c-a250-cb958115e5a2\") " pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141778    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cnibin\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovn-node-metrics-cert\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.143120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141850    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcl8m\" (UniqueName: \"kubernetes.io/projected/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-kube-api-access-zcl8m\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141872    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-socket-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-multus-certs\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141922    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-multus-certs\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.141947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/056de880-e0a6-4b20-805f-669a017b3d1d-hosts-file\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47d4b722-8199-4e8e-87bf-915e1ac409f5-tmp\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142007    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-cni-multus\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142036    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056de880-e0a6-4b20-805f-669a017b3d1d-tmp-dir\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142044    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/056de880-e0a6-4b20-805f-669a017b3d1d-hosts-file\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzswc\" (UniqueName: \"kubernetes.io/projected/47d4b722-8199-4e8e-87bf-915e1ac409f5-kube-api-access-lzswc\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-slash\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbbg\" (UniqueName: \"kubernetes.io/projected/90b722fb-cc4e-4c62-877b-db0df5c5d16c-kube-api-access-xlbbg\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-sys\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-tuned\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cni-binary-copy\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142223    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-slash\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.143896 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142268    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-daemon-config\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142273    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovnkube-config\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142295    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142319    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwdf9\" (UniqueName: \"kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9\") pod \"network-check-target-mmv7s\" (UID: \"52b495cc-bf15-4438-8762-aab455e12701\") " pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-serviceca\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dbdb\" (UniqueName: \"kubernetes.io/projected/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-kube-api-access-7dbdb\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142457    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-cni-bin\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a957db89-5022-4877-aa05-3713401881ed-host-slash\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142506    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-system-cni-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-etc-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142555    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-k8s-cni-cncf-io\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-var-lib-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a957db89-5022-4877-aa05-3713401881ed-iptables-alerter-script\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142704    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-system-cni-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shnrb\" (UniqueName: \"kubernetes.io/projected/056de880-e0a6-4b20-805f-669a017b3d1d-kube-api-access-shnrb\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5c68fab-89c6-485c-a250-cb958115e5a2-konnectivity-ca\") pod \"konnectivity-agent-c7b69\" (UID: \"f5c68fab-89c6-485c-a250-cb958115e5a2\") " pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:06:19.144735 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142813    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a957db89-5022-4877-aa05-3713401881ed-host-slash\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysconfig\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142842    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/90b722fb-cc4e-4c62-877b-db0df5c5d16c-cni-binary-copy\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142778    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovnkube-script-lib\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142829    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-serviceca\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142873    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-k8s-cni-cncf-io\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142893    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-cni-bin\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142929    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-var-lib-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142961    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-etc-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.142980    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.143033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhdgp\" (UniqueName: \"kubernetes.io/projected/2cb0ab42-11a8-493c-8218-4755fb32e543-kube-api-access-rhdgp\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.143046    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.143081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-modprobe-d\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.143165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lblgh\" (UniqueName: \"kubernetes.io/projected/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-kube-api-access-lblgh\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.143193    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a957db89-5022-4877-aa05-3713401881ed-iptables-alerter-script\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:06:19.145697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.143402    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5c68fab-89c6-485c-a250-cb958115e5a2-konnectivity-ca\") pod \"konnectivity-agent-c7b69\" (UID: \"f5c68fab-89c6-485c-a250-cb958115e5a2\") " pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:06:19.146267 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.146131    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovn-node-metrics-cert\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.146267 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.146160    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5c68fab-89c6-485c-a250-cb958115e5a2-agent-certs\") pod \"konnectivity-agent-c7b69\" (UID: \"f5c68fab-89c6-485c-a250-cb958115e5a2\") " pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:06:19.199451 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.199403    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6k56\" (UniqueName: \"kubernetes.io/projected/a957db89-5022-4877-aa05-3713401881ed-kube-api-access-l6k56\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:06:19.200086 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.200064    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhdgp\" (UniqueName: \"kubernetes.io/projected/2cb0ab42-11a8-493c-8218-4755fb32e543-kube-api-access-rhdgp\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.200210 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.200089    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shnrb\" (UniqueName: \"kubernetes.io/projected/056de880-e0a6-4b20-805f-669a017b3d1d-kube-api-access-shnrb\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:06:19.200210 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.200094    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcl8m\" (UniqueName: \"kubernetes.io/projected/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-kube-api-access-zcl8m\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.200210 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.200143    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbbg\" (UniqueName: \"kubernetes.io/projected/90b722fb-cc4e-4c62-877b-db0df5c5d16c-kube-api-access-xlbbg\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.200318 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.200284    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dbdb\" (UniqueName: \"kubernetes.io/projected/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-kube-api-access-7dbdb\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:06:19.244326 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.244276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwwgj\" (UniqueName: \"kubernetes.io/projected/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-kube-api-access-xwwgj\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:19.244525 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.244353    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-systemd\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.244525 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.244402    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysctl-conf\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.244525 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.244424    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.244525 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.244501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-run\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.244525 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.244527    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-kubernetes\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.244719 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.244527    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-run\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.244719 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.244549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysctl-d\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245001 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.244968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cnibin\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.245001 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.244527    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-systemd\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245001 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245004    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-kubernetes\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245208 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.244968    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysctl-conf\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245208 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47d4b722-8199-4e8e-87bf-915e1ac409f5-tmp\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245208 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245016    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysctl-d\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245208 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245074    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cnibin\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.245208 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245132    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzswc\" (UniqueName: \"kubernetes.io/projected/47d4b722-8199-4e8e-87bf-915e1ac409f5-kube-api-access-lzswc\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245208 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.245208 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-sys\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245571 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-tuned\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245571 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cni-binary-copy\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.245571 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245299    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-sys\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245571 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.245571 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdf9\" (UniqueName: \"kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9\") pod \"network-check-target-mmv7s\" (UID: \"52b495cc-bf15-4438-8762-aab455e12701\") " pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:19.245571 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245491    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysconfig\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245571 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:19.245571 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245552    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.245571 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-modprobe-d\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245573    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lblgh\" (UniqueName: \"kubernetes.io/projected/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-kube-api-access-lblgh\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245616    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysconfig\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-lib-modules\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-var-lib-kubelet\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-system-cni-dir\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245721    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-modprobe-d\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.245673    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245748    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cni-binary-copy\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245786    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-lib-modules\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.245797    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs podName:9bcc2a31-3406-4199-b08d-1ad8817a5cc2 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:19.745777957 +0000 UTC m=+3.232817616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs") pod "network-metrics-daemon-58h48" (UID: "9bcc2a31-3406-4199-b08d-1ad8817a5cc2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-host\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245837    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-system-cni-dir\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245804    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-host\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.245967 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.245874    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-var-lib-kubelet\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.246630 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.246441    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.248035 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.248015    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-tuned\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.248128 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.248057    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47d4b722-8199-4e8e-87bf-915e1ac409f5-tmp\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.300521 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.300483    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:19.300521 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.300514    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:19.300521 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.300525    2577 projected.go:196] Error preparing data for projected volume kube-api-access-lwdf9 for pod openshift-network-diagnostics/network-check-target-mmv7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:19.300791 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.300614    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9 podName:52b495cc-bf15-4438-8762-aab455e12701 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:19.800591809 +0000 UTC m=+3.287631472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lwdf9" (UniqueName: "kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9") pod "network-check-target-mmv7s" (UID: "52b495cc-bf15-4438-8762-aab455e12701") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:19.302111 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.302090    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwwgj\" (UniqueName: \"kubernetes.io/projected/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-kube-api-access-xwwgj\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:19.302631 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.302612    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lblgh\" (UniqueName: \"kubernetes.io/projected/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-kube-api-access-lblgh\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.302746 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.302724    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzswc\" (UniqueName: \"kubernetes.io/projected/47d4b722-8199-4e8e-87bf-915e1ac409f5-kube-api-access-lzswc\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.324592 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.324510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:06:19.330315 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.330293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:06:19.335244 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:19.335215    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda957db89_5022_4877_aa05_3713401881ed.slice/crio-4ec5f69d636b214266e4afa421e5792d890a81db8f8e8f5db5cad39d0e21e8c4 WatchSource:0}: Error finding container 4ec5f69d636b214266e4afa421e5792d890a81db8f8e8f5db5cad39d0e21e8c4: Status 404 returned error can't find the container with id 4ec5f69d636b214266e4afa421e5792d890a81db8f8e8f5db5cad39d0e21e8c4
Oct 31 03:06:19.338907 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.338886    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6vx62"
Oct 31 03:06:19.340143 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:19.340121    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3a8dde_2755_44ac_9fc3_6bb5ddbaa9aa.slice/crio-2d3fe1d96745eb4e733fc9f89eebf2eda108501e387f4349ffb9579ffdfa4c56 WatchSource:0}: Error finding container 2d3fe1d96745eb4e733fc9f89eebf2eda108501e387f4349ffb9579ffdfa4c56: Status 404 returned error can't find the container with id 2d3fe1d96745eb4e733fc9f89eebf2eda108501e387f4349ffb9579ffdfa4c56
Oct 31 03:06:19.343894 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.343875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:06:19.348388 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:19.348364    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b722fb_cc4e_4c62_877b_db0df5c5d16c.slice/crio-b13e823829260663bb49df1ca87e7d3309632c077cbd5b794617aaadedfd598b WatchSource:0}: Error finding container b13e823829260663bb49df1ca87e7d3309632c077cbd5b794617aaadedfd598b: Status 404 returned error can't find the container with id b13e823829260663bb49df1ca87e7d3309632c077cbd5b794617aaadedfd598b
Oct 31 03:06:19.352568 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.352011    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:06:19.354729 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:19.354706    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056de880_e0a6_4b20_805f_669a017b3d1d.slice/crio-64491765ef85d4cabbae65ed27e83cbb6188588cf0895952d26d16d7f6071fed WatchSource:0}: Error finding container 64491765ef85d4cabbae65ed27e83cbb6188588cf0895952d26d16d7f6071fed: Status 404 returned error can't find the container with id 64491765ef85d4cabbae65ed27e83cbb6188588cf0895952d26d16d7f6071fed
Oct 31 03:06:19.361263 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:19.361243    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c68fab_89c6_485c_a250_cb958115e5a2.slice/crio-e49095abaa2842e614824b2264069e2370b832b0454c1cedf7577eb10be3f11b WatchSource:0}: Error finding container e49095abaa2842e614824b2264069e2370b832b0454c1cedf7577eb10be3f11b: Status 404 returned error can't find the container with id e49095abaa2842e614824b2264069e2370b832b0454c1cedf7577eb10be3f11b
Oct 31 03:06:19.361879 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.361852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:06:19.371172 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:19.371148    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb0ab42_11a8_493c_8218_4755fb32e543.slice/crio-622aafbf0dd8f040baf632757fa42b94b6b3f2b35c29f708587332a38c5e5f82 WatchSource:0}: Error finding container 622aafbf0dd8f040baf632757fa42b94b6b3f2b35c29f708587332a38c5e5f82: Status 404 returned error can't find the container with id 622aafbf0dd8f040baf632757fa42b94b6b3f2b35c29f708587332a38c5e5f82
Oct 31 03:06:19.375515 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.375499    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:06:19.384623 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:19.384601    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d4b722_8199_4e8e_87bf_915e1ac409f5.slice/crio-f2f491512019e5310fb6ac8ad405272b7b24188e6dfc8982769d68a53505b146 WatchSource:0}: Error finding container f2f491512019e5310fb6ac8ad405272b7b24188e6dfc8982769d68a53505b146: Status 404 returned error can't find the container with id f2f491512019e5310fb6ac8ad405272b7b24188e6dfc8982769d68a53505b146
Oct 31 03:06:19.387844 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.387825    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:19.393665 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.393647    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:06:19.396902 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.396885    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:06:19.397344 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:19.397323    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda5dcc5_e5c6_4e0a_9acb_6191475f552c.slice/crio-9b516d30a0d2610ab2354787c4667cf4555c8a1ce3ef2c105dff044a87d227c6 WatchSource:0}: Error finding container 9b516d30a0d2610ab2354787c4667cf4555c8a1ce3ef2c105dff044a87d227c6: Status 404 returned error can't find the container with id 9b516d30a0d2610ab2354787c4667cf4555c8a1ce3ef2c105dff044a87d227c6
Oct 31 03:06:19.405305 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:19.405284    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf1fa41_a03b_4a64_ba80_4c7a2825e259.slice/crio-876a361fa14fd14406a22cd3e2e4150563d7a08a7131204279c32ab1bb2fc1fc WatchSource:0}: Error finding container 876a361fa14fd14406a22cd3e2e4150563d7a08a7131204279c32ab1bb2fc1fc: Status 404 returned error can't find the container with id 876a361fa14fd14406a22cd3e2e4150563d7a08a7131204279c32ab1bb2fc1fc
Oct 31 03:06:19.649328 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.649239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:19.649478 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.649412    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:19.649523 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.649479    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret podName:f714c9b3-ccf2-4dab-9220-321c5b4258d0 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:20.649464727 +0000 UTC m=+4.136504378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret") pod "global-pull-secret-syncer-ndfgt" (UID: "f714c9b3-ccf2-4dab-9220-321c5b4258d0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:19.750539 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.750499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:19.750743 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.750629    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:19.750743 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.750696    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs podName:9bcc2a31-3406-4199-b08d-1ad8817a5cc2 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:20.75067846 +0000 UTC m=+4.237718109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs") pod "network-metrics-daemon-58h48" (UID: "9bcc2a31-3406-4199-b08d-1ad8817a5cc2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:19.851210 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:19.851175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdf9\" (UniqueName: \"kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9\") pod \"network-check-target-mmv7s\" (UID: \"52b495cc-bf15-4438-8762-aab455e12701\") " pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:19.851404 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.851345    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:19.851404 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.851372    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:19.851404 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.851386    2577 projected.go:196] Error preparing data for projected volume kube-api-access-lwdf9 for pod openshift-network-diagnostics/network-check-target-mmv7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:19.851503 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:19.851445    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9 podName:52b495cc-bf15-4438-8762-aab455e12701 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:20.851425994 +0000 UTC m=+4.338465640 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lwdf9" (UniqueName: "kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9") pod "network-check-target-mmv7s" (UID: "52b495cc-bf15-4438-8762-aab455e12701") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:20.071806 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:20.071713    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brs5l" event={"ID":"fdf1fa41-a03b-4a64-ba80-4c7a2825e259","Type":"ContainerStarted","Data":"876a361fa14fd14406a22cd3e2e4150563d7a08a7131204279c32ab1bb2fc1fc"}
Oct 31 03:06:20.073048 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:20.073022    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs" event={"ID":"cda5dcc5-e5c6-4e0a-9acb-6191475f552c","Type":"ContainerStarted","Data":"9b516d30a0d2610ab2354787c4667cf4555c8a1ce3ef2c105dff044a87d227c6"}
Oct 31 03:06:20.074158 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:20.074133    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c7b69" event={"ID":"f5c68fab-89c6-485c-a250-cb958115e5a2","Type":"ContainerStarted","Data":"e49095abaa2842e614824b2264069e2370b832b0454c1cedf7577eb10be3f11b"}
Oct 31 03:06:20.075217 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:20.075190    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4fpkl" event={"ID":"056de880-e0a6-4b20-805f-669a017b3d1d","Type":"ContainerStarted","Data":"64491765ef85d4cabbae65ed27e83cbb6188588cf0895952d26d16d7f6071fed"}
Oct 31 03:06:20.076320 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:20.076297    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6vx62" event={"ID":"90b722fb-cc4e-4c62-877b-db0df5c5d16c","Type":"ContainerStarted","Data":"b13e823829260663bb49df1ca87e7d3309632c077cbd5b794617aaadedfd598b"}
Oct 31 03:06:20.077421 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:20.077401    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sdwgq" event={"ID":"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa","Type":"ContainerStarted","Data":"2d3fe1d96745eb4e733fc9f89eebf2eda108501e387f4349ffb9579ffdfa4c56"}
Oct 31 03:06:20.078442 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:20.078424    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fwmjk" event={"ID":"47d4b722-8199-4e8e-87bf-915e1ac409f5","Type":"ContainerStarted","Data":"f2f491512019e5310fb6ac8ad405272b7b24188e6dfc8982769d68a53505b146"}
Oct 31 03:06:20.079499 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:20.079475    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh" event={"ID":"2cb0ab42-11a8-493c-8218-4755fb32e543","Type":"ContainerStarted","Data":"622aafbf0dd8f040baf632757fa42b94b6b3f2b35c29f708587332a38c5e5f82"}
Oct 31 03:06:20.080590 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:20.080572    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-28rvl" event={"ID":"a957db89-5022-4877-aa05-3713401881ed","Type":"ContainerStarted","Data":"4ec5f69d636b214266e4afa421e5792d890a81db8f8e8f5db5cad39d0e21e8c4"}
Oct 31 03:06:20.118605 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:20.118568    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:01:19 +0000 UTC" deadline="2028-05-29 13:27:33.699549061 +0000 UTC"
Oct 31 03:06:20.118605 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:20.118597    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13834h21m13.580956213s"
Oct 31 03:06:20.658490 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:20.658377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:20.658692 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:20.658529    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:20.658692 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:20.658600    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret podName:f714c9b3-ccf2-4dab-9220-321c5b4258d0 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:22.658581547 +0000 UTC m=+6.145621195 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret") pod "global-pull-secret-syncer-ndfgt" (UID: "f714c9b3-ccf2-4dab-9220-321c5b4258d0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:20.759685 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:20.759638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:20.759933 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:20.759895    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:20.760044 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:20.759969    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs podName:9bcc2a31-3406-4199-b08d-1ad8817a5cc2 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:22.759949238 +0000 UTC m=+6.246988886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs") pod "network-metrics-daemon-58h48" (UID: "9bcc2a31-3406-4199-b08d-1ad8817a5cc2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:20.860715 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:20.860674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdf9\" (UniqueName: \"kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9\") pod \"network-check-target-mmv7s\" (UID: \"52b495cc-bf15-4438-8762-aab455e12701\") " pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:20.860937 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:20.860903    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:20.860937 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:20.860924    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:20.860937 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:20.860936    2577 projected.go:196] Error preparing data for projected volume kube-api-access-lwdf9 for pod openshift-network-diagnostics/network-check-target-mmv7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:20.861122 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:20.861000    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9 podName:52b495cc-bf15-4438-8762-aab455e12701 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:22.860980391 +0000 UTC m=+6.348020052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lwdf9" (UniqueName: "kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9") pod "network-check-target-mmv7s" (UID: "52b495cc-bf15-4438-8762-aab455e12701") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:21.067253 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:21.067177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:21.067420 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:21.067341    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:21.067492 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:21.067430    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:21.067539 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:21.067507    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:21.067595 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:21.067564    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:21.067650 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:21.067630    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:21.118933 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:21.118858    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:01:19 +0000 UTC" deadline="2028-07-21 17:24:01.199438962 +0000 UTC"
Oct 31 03:06:21.118933 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:21.118929    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15110h17m40.080513698s"
Oct 31 03:06:22.683211 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:22.683170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:22.683937 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:22.683378    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:22.683937 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:22.683446    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret podName:f714c9b3-ccf2-4dab-9220-321c5b4258d0 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:26.683425009 +0000 UTC m=+10.170464679 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret") pod "global-pull-secret-syncer-ndfgt" (UID: "f714c9b3-ccf2-4dab-9220-321c5b4258d0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:22.784302 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:22.784257    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:22.784531 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:22.784490    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:22.784616 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:22.784554    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs podName:9bcc2a31-3406-4199-b08d-1ad8817a5cc2 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:26.784535141 +0000 UTC m=+10.271574802 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs") pod "network-metrics-daemon-58h48" (UID: "9bcc2a31-3406-4199-b08d-1ad8817a5cc2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:22.886426 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:22.885762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdf9\" (UniqueName: \"kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9\") pod \"network-check-target-mmv7s\" (UID: \"52b495cc-bf15-4438-8762-aab455e12701\") " pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:22.886426 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:22.885972    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:22.886426 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:22.885994    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:22.886426 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:22.886004    2577 projected.go:196] Error preparing data for projected volume kube-api-access-lwdf9 for pod openshift-network-diagnostics/network-check-target-mmv7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:22.886426 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:22.886053    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9 podName:52b495cc-bf15-4438-8762-aab455e12701 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:26.886038386 +0000 UTC m=+10.373078046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lwdf9" (UniqueName: "kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9") pod "network-check-target-mmv7s" (UID: "52b495cc-bf15-4438-8762-aab455e12701") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:23.066055 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:23.065966    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:23.066225 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:23.066116    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:23.066518 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:23.066496    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:23.066620 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:23.066599    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:23.066683 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:23.066659    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:23.066771 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:23.066752    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:24.124780 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:24.124650    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal" event={"ID":"584715508ebcb798df8a2f82469ae3b6","Type":"ContainerStarted","Data":"e8991270f7e891d11886dc413f42b5d90d0e165389ed2b8530e1d93a7b23351a"}
Oct 31 03:06:25.066224 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:25.066144    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:25.066381 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:25.066290    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:25.066717 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:25.066694    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:25.066817 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:25.066797    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:25.067291 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:25.067259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:25.067420 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:25.067398    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:25.134598 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:25.134555    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d194eddb65e9ecc3707084f3a0166db" containerID="b239b72aa94ef50bd34a6618c5559ec7477d41b2365777bd8c89e5dbf4403910" exitCode=0
Oct 31 03:06:25.135159 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:25.134662    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal" event={"ID":"9d194eddb65e9ecc3707084f3a0166db","Type":"ContainerDied","Data":"b239b72aa94ef50bd34a6618c5559ec7477d41b2365777bd8c89e5dbf4403910"}
Oct 31 03:06:25.140120 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:25.140084    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" containerID="9655ec86740d5d15396b58e641380c155a1dadc3baff0e06162bb478b0814200" exitCode=0
Oct 31 03:06:25.140248 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:25.140183    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brs5l" event={"ID":"fdf1fa41-a03b-4a64-ba80-4c7a2825e259","Type":"ContainerDied","Data":"9655ec86740d5d15396b58e641380c155a1dadc3baff0e06162bb478b0814200"}
Oct 31 03:06:25.146365 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:25.146336    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c7b69" event={"ID":"f5c68fab-89c6-485c-a250-cb958115e5a2","Type":"ContainerStarted","Data":"87361474e7ba16777cc634487b2073c57ec19c4d3e209539b0119f6448737410"}
Oct 31 03:06:25.249470 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:25.249401    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal" podStartSLOduration=6.249379381 podStartE2EDuration="6.249379381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:24.239982273 +0000 UTC m=+7.727021944" watchObservedRunningTime="2026-10-31 03:06:25.249379381 +0000 UTC m=+8.736419054"
Oct 31 03:06:26.154987 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:26.154255    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal" event={"ID":"9d194eddb65e9ecc3707084f3a0166db","Type":"ContainerStarted","Data":"68a8a97288afcc8a244282947d6db9e9e566ae7cdc2c1c105662067de40cf0b4"}
Oct 31 03:06:26.158165 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:26.158092    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sdwgq" event={"ID":"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa","Type":"ContainerStarted","Data":"ca25854047255841d475a98972fab694f55bafa8e15627b0d95f39cea197e7b7"}
Oct 31 03:06:26.287104 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:26.287021    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c7b69" podStartSLOduration=3.992879934 podStartE2EDuration="9.286998042s" totalImagesPullingTime="5.294118108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:25.482764632 +0000 UTC m=+8.969804303" watchObservedRunningTime="2026-10-31 03:06:26.286998042 +0000 UTC m=+9.774037724"
Oct 31 03:06:26.288354 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:26.287738    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal" podStartSLOduration=7.28772594 podStartE2EDuration="7.28772594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:26.287603126 +0000 UTC m=+9.774642818" watchObservedRunningTime="2026-10-31 03:06:26.28772594 +0000 UTC m=+9.774765610"
Oct 31 03:06:26.728568 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:26.728474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:26.728746 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:26.728717    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:26.728879 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:26.728863    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret podName:f714c9b3-ccf2-4dab-9220-321c5b4258d0 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:34.728798623 +0000 UTC m=+18.215838278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret") pod "global-pull-secret-syncer-ndfgt" (UID: "f714c9b3-ccf2-4dab-9220-321c5b4258d0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:26.830137 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:26.830094    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:26.830316 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:26.830258    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:26.830380 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:26.830326    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs podName:9bcc2a31-3406-4199-b08d-1ad8817a5cc2 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:34.83030582 +0000 UTC m=+18.317345481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs") pod "network-metrics-daemon-58h48" (UID: "9bcc2a31-3406-4199-b08d-1ad8817a5cc2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:26.931824 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:26.931201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdf9\" (UniqueName: \"kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9\") pod \"network-check-target-mmv7s\" (UID: \"52b495cc-bf15-4438-8762-aab455e12701\") " pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:26.931824 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:26.931419    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:26.931824 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:26.931441    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:26.931824 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:26.931463    2577 projected.go:196] Error preparing data for projected volume kube-api-access-lwdf9 for pod openshift-network-diagnostics/network-check-target-mmv7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:26.931824 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:26.931509    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9 podName:52b495cc-bf15-4438-8762-aab455e12701 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:34.931497034 +0000 UTC m=+18.418536694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lwdf9" (UniqueName: "kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9") pod "network-check-target-mmv7s" (UID: "52b495cc-bf15-4438-8762-aab455e12701") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:27.063644 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:27.063563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:27.063840 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:27.063668    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:27.063983 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:27.063961    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:27.064056 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:27.064041    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:27.064158 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:27.064141    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:27.064278 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:27.064257    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:27.163589 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:27.163544    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh" event={"ID":"2cb0ab42-11a8-493c-8218-4755fb32e543","Type":"ContainerStarted","Data":"d66f190fd3bfe8e236a5d98d35833472c1dcc98a4764c6ee7821247b83da66bc"}
Oct 31 03:06:28.897626 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:28.897604    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 03:06:28.990218 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:28.990068    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:06:28.897623009Z","UUID":"0b268776-f799-4b99-b863-08ad52cfabd1","Handler":null,"Name":"","Endpoint":""}
Oct 31 03:06:28.991832 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:28.991807    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 03:06:28.991950 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:28.991849    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 03:06:29.064935 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:29.064900    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:29.065093 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:29.065033    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:29.065164 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:29.065114    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:29.065214 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:29.065199    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:29.066046 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:29.066023    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:29.066157 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:29.066116    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:29.172197 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:29.172160    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4fpkl" event={"ID":"056de880-e0a6-4b20-805f-669a017b3d1d","Type":"ContainerStarted","Data":"8f7efc89c18df54230217ad1c9d5f7fe5948acd88933d874820f59eae1f95796"}
Oct 31 03:06:29.174298 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:29.174267    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6vx62" event={"ID":"90b722fb-cc4e-4c62-877b-db0df5c5d16c","Type":"ContainerStarted","Data":"7111a0c93028da1d2d8d0119bd1620b4eb533f5a203d043fcd027290022c871e"}
Oct 31 03:06:29.176800 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:29.176758    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh" event={"ID":"2cb0ab42-11a8-493c-8218-4755fb32e543","Type":"ContainerStarted","Data":"250f408c296bcd086f8b37945e29c5b3d1ae595cf565fc64233e250c8f9eb9ac"}
Oct 31 03:06:29.278299 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:29.278013    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sdwgq" podStartSLOduration=5.705175946 podStartE2EDuration="12.277999378s" totalImagesPullingTime="6.572823432s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:26.402716302 +0000 UTC m=+9.889755971" watchObservedRunningTime="2026-10-31 03:06:29.277999378 +0000 UTC m=+12.765039043"
Oct 31 03:06:29.386488 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:29.386426    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-6vx62" podStartSLOduration=3.256188821 podStartE2EDuration="12.38640732s" totalImagesPullingTime="9.130218499s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:29.386300725 +0000 UTC m=+12.873340390" watchObservedRunningTime="2026-10-31 03:06:29.38640732 +0000 UTC m=+12.873446990"
Oct 31 03:06:29.387115 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:29.387072    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-4fpkl" podStartSLOduration=2.741723427 podStartE2EDuration="12.387061155s" totalImagesPullingTime="9.645337728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:29.279173821 +0000 UTC m=+12.766213486" watchObservedRunningTime="2026-10-31 03:06:29.387061155 +0000 UTC m=+12.874100826"
Oct 31 03:06:29.502917 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:29.502253    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:06:29.502917 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:29.502731    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:06:30.182223 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:30.182177    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" containerID="308c480d5b47124a0fa0c9fa4415089f36cbaf5b44f19c85f442ca49a7d4fbd7" exitCode=0
Oct 31 03:06:30.182708 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:30.182685    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brs5l" event={"ID":"fdf1fa41-a03b-4a64-ba80-4c7a2825e259","Type":"ContainerDied","Data":"308c480d5b47124a0fa0c9fa4415089f36cbaf5b44f19c85f442ca49a7d4fbd7"}
Oct 31 03:06:31.066650 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:31.066624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:31.066751 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:31.066656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:31.066751 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:31.066723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:31.066751 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:31.066737    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:31.066918 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:31.066807    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:31.066918 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:31.066884    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:31.186511 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:31.186465    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fwmjk" event={"ID":"47d4b722-8199-4e8e-87bf-915e1ac409f5","Type":"ContainerStarted","Data":"2ca81bee609c7ac471b0a112351d2e376016a7f739076907973c1009475f45f8"}
Oct 31 03:06:31.188681 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:31.188644    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-28rvl" event={"ID":"a957db89-5022-4877-aa05-3713401881ed","Type":"ContainerStarted","Data":"4d4d43bd7a7eb171848b0cfa43676ae2f117741e40d071a6cb40fec9723aad82"}
Oct 31 03:06:31.293798 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:31.293682    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fwmjk" podStartSLOduration=1.692588137 podStartE2EDuration="13.2936697s" totalImagesPullingTime="11.601081563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:31.2934004 +0000 UTC m=+14.780440066" watchObservedRunningTime="2026-10-31 03:06:31.2936697 +0000 UTC m=+14.780709366"
Oct 31 03:06:32.193593 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:32.193429    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" containerID="080ecbcedf4234973f699feb56de24640fe2a48f208da8c94b64e272e52752f6" exitCode=0
Oct 31 03:06:32.193593 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:32.193473    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brs5l" event={"ID":"fdf1fa41-a03b-4a64-ba80-4c7a2825e259","Type":"ContainerDied","Data":"080ecbcedf4234973f699feb56de24640fe2a48f208da8c94b64e272e52752f6"}
Oct 31 03:06:32.305076 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:32.305018    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-28rvl" podStartSLOduration=5.872426525 podStartE2EDuration="15.305003656s" totalImagesPullingTime="9.432577131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:31.397585864 +0000 UTC m=+14.884625533" watchObservedRunningTime="2026-10-31 03:06:32.305003656 +0000 UTC m=+15.792043319"
Oct 31 03:06:33.062947 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:33.062860    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:33.062947 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:33.062917    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:33.063168 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:33.062926    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:33.063168 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:33.063023    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:33.063258 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:33.063152    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:33.063303 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:33.063264    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:33.197616 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:33.197579    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh" event={"ID":"2cb0ab42-11a8-493c-8218-4755fb32e543","Type":"ContainerStarted","Data":"72d48af6fad01930dd430e50744760c655284c810b372cf7818c2fbac84e593d"}
Oct 31 03:06:33.306566 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:33.306310    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh" podStartSLOduration=3.507742067 podStartE2EDuration="16.306288964s" totalImagesPullingTime="12.798546897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:33.30578267 +0000 UTC m=+16.792822367" watchObservedRunningTime="2026-10-31 03:06:33.306288964 +0000 UTC m=+16.793328631"
Oct 31 03:06:34.202775 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:34.202687    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" containerID="e15b951ac0a0c8746d765137d12b4a19536c4d5c5833ddfaf19d74e8e4fb0fd8" exitCode=0
Oct 31 03:06:34.202775 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:34.202756    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brs5l" event={"ID":"fdf1fa41-a03b-4a64-ba80-4c7a2825e259","Type":"ContainerDied","Data":"e15b951ac0a0c8746d765137d12b4a19536c4d5c5833ddfaf19d74e8e4fb0fd8"}
Oct 31 03:06:34.791739 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:34.791701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:34.791931 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:34.791834    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:34.791931 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:34.791885    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret podName:f714c9b3-ccf2-4dab-9220-321c5b4258d0 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:50.791873067 +0000 UTC m=+34.278912710 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret") pod "global-pull-secret-syncer-ndfgt" (UID: "f714c9b3-ccf2-4dab-9220-321c5b4258d0") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:06:34.892322 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:34.892274    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:34.892461 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:34.892416    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:34.892517 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:34.892475    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs podName:9bcc2a31-3406-4199-b08d-1ad8817a5cc2 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:50.892459853 +0000 UTC m=+34.379499500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs") pod "network-metrics-daemon-58h48" (UID: "9bcc2a31-3406-4199-b08d-1ad8817a5cc2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:06:34.993230 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:34.993192    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdf9\" (UniqueName: \"kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9\") pod \"network-check-target-mmv7s\" (UID: \"52b495cc-bf15-4438-8762-aab455e12701\") " pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:34.993373 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:34.993349    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:06:34.993373 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:34.993367    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:06:34.993438 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:34.993377    2577 projected.go:196] Error preparing data for projected volume kube-api-access-lwdf9 for pod openshift-network-diagnostics/network-check-target-mmv7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:34.993438 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:34.993426    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9 podName:52b495cc-bf15-4438-8762-aab455e12701 nodeName:}" failed. No retries permitted until 2026-10-31 03:06:50.993411268 +0000 UTC m=+34.480450920 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lwdf9" (UniqueName: "kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9") pod "network-check-target-mmv7s" (UID: "52b495cc-bf15-4438-8762-aab455e12701") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:06:35.063016 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:35.062941    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:35.063016 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:35.063000    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:35.063016 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:35.062998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:35.063247 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:35.063099    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:35.063247 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:35.063211    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:35.063350 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:35.063294    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:36.210683 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:36.210513    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:06:36.211173 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:36.211016    2577 generic.go:356] "Generic (PLEG): container finished" podID="cda5dcc5-e5c6-4e0a-9acb-6191475f552c" containerID="e89a6744ed4fe5f9ef26d0cc49124150bd9a6f0792d62f25452d975faa0505ed" exitCode=1
Oct 31 03:06:36.211173 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:36.211066    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs" event={"ID":"cda5dcc5-e5c6-4e0a-9acb-6191475f552c","Type":"ContainerStarted","Data":"ab4e64eef6d01222cd32839202245e0de050e58005fc2f12dd076bcbf9465ea8"}
Oct 31 03:06:36.211173 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:36.211122    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs" event={"ID":"cda5dcc5-e5c6-4e0a-9acb-6191475f552c","Type":"ContainerStarted","Data":"283220037d7f0c0a72acf60677ab35c2e06af2152b250723baeba22c06b78518"}
Oct 31 03:06:36.211173 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:36.211158    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs" event={"ID":"cda5dcc5-e5c6-4e0a-9acb-6191475f552c","Type":"ContainerStarted","Data":"8f46a3991a61733a789835f1e97f0a2cddf88cfe4a298f0ec3dcbe2ec2a9e696"}
Oct 31 03:06:36.211366 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:36.211200    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs" event={"ID":"cda5dcc5-e5c6-4e0a-9acb-6191475f552c","Type":"ContainerStarted","Data":"a4d37dcc2ae83964eedacca18ecb1908d56e352790ca9eb20e33cff1ddeb61e8"}
Oct 31 03:06:36.211366 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:36.211233    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs" event={"ID":"cda5dcc5-e5c6-4e0a-9acb-6191475f552c","Type":"ContainerDied","Data":"e89a6744ed4fe5f9ef26d0cc49124150bd9a6f0792d62f25452d975faa0505ed"}
Oct 31 03:06:36.211366 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:36.211268    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs" event={"ID":"cda5dcc5-e5c6-4e0a-9acb-6191475f552c","Type":"ContainerStarted","Data":"61ea865664d4136ffce5bec54bdbb300e990702fc4f88abf50fd1171afd0a286"}
Oct 31 03:06:37.063421 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:37.063383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:37.063553 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:37.063474    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:37.063553 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:37.063524    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:37.063553 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:37.063538    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:37.063663 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:37.063569    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:37.063693 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:37.063660    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:37.215756 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:37.215680    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" containerID="e70e374d67099e25ab8a1b859e3915ae7119c2476703454dacc84a61b526eb81" exitCode=0
Oct 31 03:06:37.215756 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:37.215733    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brs5l" event={"ID":"fdf1fa41-a03b-4a64-ba80-4c7a2825e259","Type":"ContainerDied","Data":"e70e374d67099e25ab8a1b859e3915ae7119c2476703454dacc84a61b526eb81"}
Oct 31 03:06:38.220353 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:38.220323    2577 generic.go:356] "Generic (PLEG): container finished" podID="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" containerID="0cd137a0436c0714916c96d907fe0e464a786c195f0d53dfcd7965fda9a6d07f" exitCode=0
Oct 31 03:06:38.220901 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:38.220407    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brs5l" event={"ID":"fdf1fa41-a03b-4a64-ba80-4c7a2825e259","Type":"ContainerDied","Data":"0cd137a0436c0714916c96d907fe0e464a786c195f0d53dfcd7965fda9a6d07f"}
Oct 31 03:06:38.223405 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:38.223353    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:06:38.223691 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:38.223673    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs" event={"ID":"cda5dcc5-e5c6-4e0a-9acb-6191475f552c","Type":"ContainerStarted","Data":"9a42cbf0cf4efb1c8d4b6975e600c3e84682a4dc8ee55ad3487452e5bb40d5b7"}
Oct 31 03:06:39.065886 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:39.065857    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:39.066058 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:39.065914    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:39.066058 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:39.065927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:39.066058 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:39.066000    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:39.066165 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:39.066057    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:39.066165 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:39.066118    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:39.228917 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:39.228883    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-brs5l" event={"ID":"fdf1fa41-a03b-4a64-ba80-4c7a2825e259","Type":"ContainerStarted","Data":"bc97394f047c668be1f712ecac7d2310f9b1a1fb39da725dd655f73189b673c2"}
Oct 31 03:06:39.432153 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:39.432070    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-brs5l" podStartSLOduration=6.862951554 podStartE2EDuration="21.432056491s" totalImagesPullingTime="14.569104937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:39.431715698 +0000 UTC m=+22.918755367" watchObservedRunningTime="2026-10-31 03:06:39.432056491 +0000 UTC m=+22.919096157"
Oct 31 03:06:40.234005 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:40.233837    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:06:40.234536 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:40.234338    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs" event={"ID":"cda5dcc5-e5c6-4e0a-9acb-6191475f552c","Type":"ContainerStarted","Data":"7dd085ec6306482c00ad3ab5773cebf466f0ff432a1d4b224f5fe7ab25e5ef0b"}
Oct 31 03:06:40.234932 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:40.234912    2577 scope.go:122] "RemoveContainer" containerID="e89a6744ed4fe5f9ef26d0cc49124150bd9a6f0792d62f25452d975faa0505ed"
Oct 31 03:06:41.063325 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:41.063288    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:41.063325 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:41.063310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:41.063566 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:41.063297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:41.063566 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:41.063416    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:41.063566 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:41.063532    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:41.063706 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:41.063608    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:41.240030 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:41.240000    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:06:41.240457 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:41.240319    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs" event={"ID":"cda5dcc5-e5c6-4e0a-9acb-6191475f552c","Type":"ContainerStarted","Data":"b8e1d89f95bd16aedc5a8e85e6940a1566f57241e3e5ff4a5e807ab9d0aa7b79"}
Oct 31 03:06:41.240710 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:41.240683    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:41.240853 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:41.240758    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:41.240853 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:41.240828    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:41.255710 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:41.255683    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:41.255854 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:41.255811    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:06:41.494750 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:41.494658    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs" podStartSLOduration=7.71141657 podStartE2EDuration="23.494644731s" totalImagesPullingTime="15.783228161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:41.448188213 +0000 UTC m=+24.935227881" watchObservedRunningTime="2026-10-31 03:06:41.494644731 +0000 UTC m=+24.981684397"
Oct 31 03:06:42.369920 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:42.369776    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mmv7s"]
Oct 31 03:06:42.369920 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:42.369918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:42.370489 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:42.370017    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:42.370489 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:42.370473    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-58h48"]
Oct 31 03:06:42.370565 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:42.370557    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:42.370672 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:42.370650    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:42.371832 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:42.371809    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ndfgt"]
Oct 31 03:06:42.371960 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:42.371890    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:42.372024 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:42.372006    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:43.079497 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:43.079463    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:43.645649 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:43.645615    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:43.893386 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:43.893352    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:44.063613 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:44.063532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:44.063778 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:44.063655    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:44.063778 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:44.063661    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:44.063778 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:44.063687    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:44.063918 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:44.063773    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:44.063918 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:44.063867    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:44.951170 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:44.951140    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:45.154410 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:45.154376    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:46.063482 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:46.063317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:46.063862 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:46.063359    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:46.063862 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:46.063573    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:46.063862 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:46.063648    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:46.063862 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:46.063362    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:46.063862 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:46.063725    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:46.372288 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:46.372200    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:06:46.372458 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:46.372317    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:06:46.372886 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:46.372860    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:06:46.617424 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:46.617395    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:47.008299 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:47.008264    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:47.207137 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:47.207094    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:47.604310 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:47.604276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:47.855619 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:47.855537    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:48.056552 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.056525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:48.063477 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.063455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:48.063600 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.063531    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:48.063640 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.063621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:48.063640 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:48.063626    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-58h48" podUID="9bcc2a31-3406-4199-b08d-1ad8817a5cc2"
Oct 31 03:06:48.063707 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:48.063689    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mmv7s" podUID="52b495cc-bf15-4438-8762-aab455e12701"
Oct 31 03:06:48.063809 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:48.063771    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ndfgt" podUID="f714c9b3-ccf2-4dab-9220-321c5b4258d0"
Oct 31 03:06:48.259509 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.259425    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:48.301297 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.301275    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeReady"
Oct 31 03:06:48.301452 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.301392    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 03:06:48.407041 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.407009    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-whz62"]
Oct 31 03:06:48.410729 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.410710    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8z98n"]
Oct 31 03:06:48.410857 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.410839    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:48.414698 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.414677    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p5zp9"]
Oct 31 03:06:48.414809 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.414783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:48.418349 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.418333    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p5zp9"
Oct 31 03:06:48.453601 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.453578    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-whz62"]
Oct 31 03:06:48.460535 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.460514    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 03:06:48.460711 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.460691    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 03:06:48.460778 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.460719    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wjqqx\""
Oct 31 03:06:48.465474 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.465457    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 03:06:48.466996 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.466975    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8z98n"]
Oct 31 03:06:48.487520 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.487486    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:48.487659 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.487551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657qr\" (UniqueName: \"kubernetes.io/projected/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-kube-api-access-657qr\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:48.487719 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.487678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-config-volume\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:48.487769 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.487739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-tmp-dir\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:48.523039 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.522962    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p5zp9"]
Oct 31 03:06:48.561725 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.561694    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 03:06:48.561956 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.561936    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 03:06:48.562018 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.561971    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 03:06:48.562139 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.562110    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 03:06:48.562139 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.562112    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 03:06:48.562286 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.562123    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sc79n\""
Oct 31 03:06:48.562286 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.562157    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 03:06:48.562286 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.562200    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m6gdv\""
Oct 31 03:06:48.588182 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.588149    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-config-volume\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:48.588344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.588187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-crio-socket\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:48.588344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.588212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:48.588344 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.588234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knqzf\" (UniqueName: \"kubernetes.io/projected/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-api-access-knqzf\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:48.588513 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.588388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-tmp-dir\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:48.588513 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.588467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-data-volume\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:48.588513 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.588505    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:48.588666 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.588533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:48.588666 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.588563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltgxk\" (UniqueName: \"kubernetes.io/projected/8b5e3f9c-61e6-4468-9891-afdeb408caf7-kube-api-access-ltgxk\") pod \"ingress-canary-p5zp9\" (UID: \"8b5e3f9c-61e6-4468-9891-afdeb408caf7\") " pod="openshift-ingress-canary/ingress-canary-p5zp9"
Oct 31 03:06:48.588666 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:48.588629    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:06:48.588666 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.588649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b5e3f9c-61e6-4468-9891-afdeb408caf7-cert\") pod \"ingress-canary-p5zp9\" (UID: \"8b5e3f9c-61e6-4468-9891-afdeb408caf7\") " pod="openshift-ingress-canary/ingress-canary-p5zp9"
Oct 31 03:06:48.588870 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.588668    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-tmp-dir\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:48.588870 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.588673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657qr\" (UniqueName: \"kubernetes.io/projected/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-kube-api-access-657qr\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:48.588870 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:48.588698    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls podName:569a2ae0-d9c0-4f18-a27c-14167c5ccdcb nodeName:}" failed. No retries permitted until 2026-10-31 03:06:49.088680827 +0000 UTC m=+32.575720483 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls") pod "dns-default-whz62" (UID: "569a2ae0-d9c0-4f18-a27c-14167c5ccdcb") : secret "dns-default-metrics-tls" not found
Oct 31 03:06:48.589052 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.588969    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-config-volume\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:48.644373 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.644338    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-657qr\" (UniqueName: \"kubernetes.io/projected/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-kube-api-access-657qr\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:48.689416 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.689386    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-data-volume\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:48.689557 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.689433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:48.689557 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.689461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltgxk\" (UniqueName: \"kubernetes.io/projected/8b5e3f9c-61e6-4468-9891-afdeb408caf7-kube-api-access-ltgxk\") pod \"ingress-canary-p5zp9\" (UID: \"8b5e3f9c-61e6-4468-9891-afdeb408caf7\") " pod="openshift-ingress-canary/ingress-canary-p5zp9"
Oct 31 03:06:48.689557 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.689487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b5e3f9c-61e6-4468-9891-afdeb408caf7-cert\") pod \"ingress-canary-p5zp9\" (UID: \"8b5e3f9c-61e6-4468-9891-afdeb408caf7\") " pod="openshift-ingress-canary/ingress-canary-p5zp9"
Oct 31 03:06:48.689671 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.689581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-crio-socket\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:48.689671 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.689601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:48.689671 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.689619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knqzf\" (UniqueName: \"kubernetes.io/projected/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-api-access-knqzf\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:48.689671 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:48.689621    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:48.689865 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:48.689697    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls podName:fdc104e8-81dd-4bef-8dee-e28bf5c7a10d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:49.189676062 +0000 UTC m=+32.676715726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8z98n" (UID: "fdc104e8-81dd-4bef-8dee-e28bf5c7a10d") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:48.689865 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:48.689757    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 03:06:48.689865 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.689773    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-data-volume\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:48.689865 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.689775    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-crio-socket\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:48.689865 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:48.689850    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm podName:fdc104e8-81dd-4bef-8dee-e28bf5c7a10d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:49.1898356 +0000 UTC m=+32.676875264 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8z98n" (UID: "fdc104e8-81dd-4bef-8dee-e28bf5c7a10d") : configmap "kube-rbac-proxy" not found
Oct 31 03:06:48.692062 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.692041    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b5e3f9c-61e6-4468-9891-afdeb408caf7-cert\") pod \"ingress-canary-p5zp9\" (UID: \"8b5e3f9c-61e6-4468-9891-afdeb408caf7\") " pod="openshift-ingress-canary/ingress-canary-p5zp9"
Oct 31 03:06:48.745196 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.745158    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knqzf\" (UniqueName: \"kubernetes.io/projected/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-api-access-knqzf\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:48.746184 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.746160    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltgxk\" (UniqueName: \"kubernetes.io/projected/8b5e3f9c-61e6-4468-9891-afdeb408caf7-kube-api-access-ltgxk\") pod \"ingress-canary-p5zp9\" (UID: \"8b5e3f9c-61e6-4468-9891-afdeb408caf7\") " pod="openshift-ingress-canary/ingress-canary-p5zp9"
Oct 31 03:06:48.773850 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.773771    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:48.985671 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:48.985635    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:49.037762 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:49.037687    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p5zp9"
Oct 31 03:06:49.093742 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:49.093711    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:49.093904 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:49.093882    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:06:49.093979 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:49.093968    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls podName:569a2ae0-d9c0-4f18-a27c-14167c5ccdcb nodeName:}" failed. No retries permitted until 2026-10-31 03:06:50.093950107 +0000 UTC m=+33.580989764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls") pod "dns-default-whz62" (UID: "569a2ae0-d9c0-4f18-a27c-14167c5ccdcb") : secret "dns-default-metrics-tls" not found
Oct 31 03:06:49.193360 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:49.193328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:49.194314 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:49.194294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:49.194386 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:49.194334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:49.194428 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:49.194412    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 03:06:49.194491 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:49.194481    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm podName:fdc104e8-81dd-4bef-8dee-e28bf5c7a10d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:50.194466203 +0000 UTC m=+33.681505852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8z98n" (UID: "fdc104e8-81dd-4bef-8dee-e28bf5c7a10d") : configmap "kube-rbac-proxy" not found
Oct 31 03:06:49.194558 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:49.194428    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:49.194558 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:49.194556    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls podName:fdc104e8-81dd-4bef-8dee-e28bf5c7a10d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:50.194543957 +0000 UTC m=+33.681583616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8z98n" (UID: "fdc104e8-81dd-4bef-8dee-e28bf5c7a10d") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:49.416399 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:49.416324    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p5zp9"]
Oct 31 03:06:49.420118 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:49.420095    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5e3f9c_61e6_4468_9891_afdeb408caf7.slice/crio-caa2b56f9caef11de2206811cb5d6524aba33a4d8e287fd1f6583f45e991c4fd WatchSource:0}: Error finding container caa2b56f9caef11de2206811cb5d6524aba33a4d8e287fd1f6583f45e991c4fd: Status 404 returned error can't find the container with id caa2b56f9caef11de2206811cb5d6524aba33a4d8e287fd1f6583f45e991c4fd
Oct 31 03:06:49.994441 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:49.994408    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-r89cn"]
Oct 31 03:06:50.017284 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.017251    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:06:50.050805 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.050756    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-r89cn"]
Oct 31 03:06:50.063185 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.063156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:50.063329 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.063228    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:50.063329 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.063319    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:50.066366 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.066338    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 03:06:50.066516 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.066408    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 03:06:50.066516 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.066452    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 03:06:50.066694 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.066677    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 03:06:50.066767 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.066704    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jbq7d\""
Oct 31 03:06:50.102245 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.102209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3313df2c-6c0b-44d8-a356-8da87e1d620a-signing-key\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:06:50.102373 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.102331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:50.102411 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.102391    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3313df2c-6c0b-44d8-a356-8da87e1d620a-signing-cabundle\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:06:50.102453 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.102441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fptbl\" (UniqueName: \"kubernetes.io/projected/3313df2c-6c0b-44d8-a356-8da87e1d620a-kube-api-access-fptbl\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:06:50.102486 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:50.102463    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:06:50.102540 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:50.102530    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls podName:569a2ae0-d9c0-4f18-a27c-14167c5ccdcb nodeName:}" failed. No retries permitted until 2026-10-31 03:06:52.102512645 +0000 UTC m=+35.589552303 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls") pod "dns-default-whz62" (UID: "569a2ae0-d9c0-4f18-a27c-14167c5ccdcb") : secret "dns-default-metrics-tls" not found
Oct 31 03:06:50.167279 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.167248    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 03:06:50.167494 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.167471    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 03:06:50.167574 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.167476    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 03:06:50.167574 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.167531    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 03:06:50.167671 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.167609    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z58pm\""
Oct 31 03:06:50.167704 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.167683    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ffsgr\""
Oct 31 03:06:50.167743 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.167727    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-89z7n\""
Oct 31 03:06:50.202921 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.202886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:50.203082 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.202940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3313df2c-6c0b-44d8-a356-8da87e1d620a-signing-cabundle\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:06:50.203082 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.202978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fptbl\" (UniqueName: \"kubernetes.io/projected/3313df2c-6c0b-44d8-a356-8da87e1d620a-kube-api-access-fptbl\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:06:50.203082 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.203010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3313df2c-6c0b-44d8-a356-8da87e1d620a-signing-key\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:06:50.203082 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.203039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:50.203082 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:50.203058    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:50.203338 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:50.203117    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 03:06:50.203338 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:50.203128    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls podName:fdc104e8-81dd-4bef-8dee-e28bf5c7a10d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:52.203110068 +0000 UTC m=+35.690149727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8z98n" (UID: "fdc104e8-81dd-4bef-8dee-e28bf5c7a10d") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:50.203338 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:50.203172    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm podName:fdc104e8-81dd-4bef-8dee-e28bf5c7a10d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:52.20315471 +0000 UTC m=+35.690194370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8z98n" (UID: "fdc104e8-81dd-4bef-8dee-e28bf5c7a10d") : configmap "kube-rbac-proxy" not found
Oct 31 03:06:50.203707 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.203685    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3313df2c-6c0b-44d8-a356-8da87e1d620a-signing-cabundle\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:06:50.205507 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.205487    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3313df2c-6c0b-44d8-a356-8da87e1d620a-signing-key\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:06:50.256615 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.256550    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fptbl\" (UniqueName: \"kubernetes.io/projected/3313df2c-6c0b-44d8-a356-8da87e1d620a-kube-api-access-fptbl\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:06:50.263167 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.263141    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p5zp9" event={"ID":"8b5e3f9c-61e6-4468-9891-afdeb408caf7","Type":"ContainerStarted","Data":"caa2b56f9caef11de2206811cb5d6524aba33a4d8e287fd1f6583f45e991c4fd"}
Oct 31 03:06:50.298341 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.298315    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:50.335945 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.335915    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:06:50.515564 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.515496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:50.701492 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.701461    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-r89cn"]
Oct 31 03:06:50.703979 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:50.703945    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3313df2c_6c0b_44d8_a356_8da87e1d620a.slice/crio-5dd6113daff0a3d568c8b28c559270b7619b13e6d7490bbad4a71a3e5c3d9183 WatchSource:0}: Error finding container 5dd6113daff0a3d568c8b28c559270b7619b13e6d7490bbad4a71a3e5c3d9183: Status 404 returned error can't find the container with id 5dd6113daff0a3d568c8b28c559270b7619b13e6d7490bbad4a71a3e5c3d9183
Oct 31 03:06:50.806156 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.806077    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:50.808297 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.808278    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:50.819865 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.819837    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:50.906346 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.906316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:06:50.906515 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:50.906470    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 03:06:50.906554 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:50.906538    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs podName:9bcc2a31-3406-4199-b08d-1ad8817a5cc2 nodeName:}" failed. No retries permitted until 2026-10-31 03:07:22.906520976 +0000 UTC m=+66.393560638 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs") pod "network-metrics-daemon-58h48" (UID: "9bcc2a31-3406-4199-b08d-1ad8817a5cc2") : secret "metrics-daemon-secret" not found
Oct 31 03:06:50.985035 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:50.985002    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:06:51.007629 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:51.007600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdf9\" (UniqueName: \"kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9\") pod \"network-check-target-mmv7s\" (UID: \"52b495cc-bf15-4438-8762-aab455e12701\") " pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:51.010001 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:51.009984    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwdf9\" (UniqueName: \"kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9\") pod \"network-check-target-mmv7s\" (UID: \"52b495cc-bf15-4438-8762-aab455e12701\") " pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:51.035653 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:51.035629    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:51.267027 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:51.266996    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn" event={"ID":"3313df2c-6c0b-44d8-a356-8da87e1d620a","Type":"ContainerStarted","Data":"5dd6113daff0a3d568c8b28c559270b7619b13e6d7490bbad4a71a3e5c3d9183"}
Oct 31 03:06:51.297250 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:51.297219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:51.371613 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:51.371471    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ndfgt"]
Oct 31 03:06:51.376334 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:51.376299    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf714c9b3_ccf2_4dab_9220_321c5b4258d0.slice/crio-709a3960720e1f59c1ae8a7a1f387366ff04f69fa827dec3ba5f6673185ab7c8 WatchSource:0}: Error finding container 709a3960720e1f59c1ae8a7a1f387366ff04f69fa827dec3ba5f6673185ab7c8: Status 404 returned error can't find the container with id 709a3960720e1f59c1ae8a7a1f387366ff04f69fa827dec3ba5f6673185ab7c8
Oct 31 03:06:51.700169 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:51.700096    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mmv7s"]
Oct 31 03:06:51.702989 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:06:51.702953    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b495cc_bf15_4438_8762_aab455e12701.slice/crio-ab67e784a14ff154144d603e90c8fce7bff0153a910b8f931121c80a8eba8ab9 WatchSource:0}: Error finding container ab67e784a14ff154144d603e90c8fce7bff0153a910b8f931121c80a8eba8ab9: Status 404 returned error can't find the container with id ab67e784a14ff154144d603e90c8fce7bff0153a910b8f931121c80a8eba8ab9
Oct 31 03:06:52.116287 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:52.116251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:52.116443 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:52.116406    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:06:52.116483 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:52.116467    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls podName:569a2ae0-d9c0-4f18-a27c-14167c5ccdcb nodeName:}" failed. No retries permitted until 2026-10-31 03:06:56.116451067 +0000 UTC m=+39.603490716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls") pod "dns-default-whz62" (UID: "569a2ae0-d9c0-4f18-a27c-14167c5ccdcb") : secret "dns-default-metrics-tls" not found
Oct 31 03:06:52.217016 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:52.216983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:52.217169 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:52.217032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:52.217169 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:52.217137    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:52.217234 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:52.217206    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls podName:fdc104e8-81dd-4bef-8dee-e28bf5c7a10d nodeName:}" failed. No retries permitted until 2026-10-31 03:06:56.217189192 +0000 UTC m=+39.704228850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8z98n" (UID: "fdc104e8-81dd-4bef-8dee-e28bf5c7a10d") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:52.217638 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:52.217621    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:52.271091 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:52.271008    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ndfgt" event={"ID":"f714c9b3-ccf2-4dab-9220-321c5b4258d0","Type":"ContainerStarted","Data":"709a3960720e1f59c1ae8a7a1f387366ff04f69fa827dec3ba5f6673185ab7c8"}
Oct 31 03:06:52.272118 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:52.272098    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mmv7s" event={"ID":"52b495cc-bf15-4438-8762-aab455e12701","Type":"ContainerStarted","Data":"ab67e784a14ff154144d603e90c8fce7bff0153a910b8f931121c80a8eba8ab9"}
Oct 31 03:06:52.396756 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:52.396723    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4fpkl_056de880-e0a6-4b20-805f-669a017b3d1d/dns-node-resolver/0.log"
Oct 31 03:06:52.407085 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:52.407058    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:52.617600 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:52.617528    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:53.505799 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:53.505765    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:53.716866 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:53.716808    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:53.797012 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:53.796936    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sdwgq_fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa/node-ca/0.log"
Oct 31 03:06:56.143616 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:56.143577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:06:56.144112 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:56.143745    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:06:56.144112 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:56.143839    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls podName:569a2ae0-d9c0-4f18-a27c-14167c5ccdcb nodeName:}" failed. No retries permitted until 2026-10-31 03:07:04.143816001 +0000 UTC m=+47.630855644 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls") pod "dns-default-whz62" (UID: "569a2ae0-d9c0-4f18-a27c-14167c5ccdcb") : secret "dns-default-metrics-tls" not found
Oct 31 03:06:56.244466 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:56.244428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:06:56.244639 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:56.244536    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:56.244639 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:06:56.244587    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls podName:fdc104e8-81dd-4bef-8dee-e28bf5c7a10d nodeName:}" failed. No retries permitted until 2026-10-31 03:07:04.24457528 +0000 UTC m=+47.731614924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8z98n" (UID: "fdc104e8-81dd-4bef-8dee-e28bf5c7a10d") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:06:57.558128 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:57.558089    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:06:59.290721 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:59.290629    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mmv7s" event={"ID":"52b495cc-bf15-4438-8762-aab455e12701","Type":"ContainerStarted","Data":"266315e922c37878297aab51d09e3d4445fdb1133d93e262954dc659a9e24419"}
Oct 31 03:06:59.290721 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:59.290710    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:06:59.292118 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:59.292092    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p5zp9" event={"ID":"8b5e3f9c-61e6-4468-9891-afdeb408caf7","Type":"ContainerStarted","Data":"ef8527fc1a954ca3a8acd4947253a86d135ca274f859e7968dfa522b4f0372af"}
Oct 31 03:06:59.398534 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:59.398488    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mmv7s" podStartSLOduration=34.13922952 podStartE2EDuration="41.398475258s" totalImagesPullingTime="7.259245738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:59.397929158 +0000 UTC m=+42.884968825" watchObservedRunningTime="2026-10-31 03:06:59.398475258 +0000 UTC m=+42.885514923"
Oct 31 03:06:59.506245 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:06:59.506193    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p5zp9" podStartSLOduration=1.977288752 podStartE2EDuration="11.506180745s" totalImagesPullingTime="9.528891993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:06:59.505245673 +0000 UTC m=+42.992285338" watchObservedRunningTime="2026-10-31 03:06:59.506180745 +0000 UTC m=+42.993220410"
Oct 31 03:07:00.295668 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:00.295630    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn" event={"ID":"3313df2c-6c0b-44d8-a356-8da87e1d620a","Type":"ContainerStarted","Data":"c1d354909cda84d0f5ee8f1e7cd618d57a6c4228c36031a21209b7d8af0b7642"}
Oct 31 03:07:00.402758 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:00.402702    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn" podStartSLOduration=2.778172494 podStartE2EDuration="11.402686776s" totalImagesPullingTime="8.624514282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:00.401982895 +0000 UTC m=+43.889022584" watchObservedRunningTime="2026-10-31 03:07:00.402686776 +0000 UTC m=+43.889726454"
Oct 31 03:07:00.850518 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:00.850480    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:01.261906 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:01.261802    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:01.518697 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:01.518613    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:01.732620 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:01.732588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:02.525278 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:02.525241    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:02.744167 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:02.744134    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:02.941801 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:02.941711    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:03.148806 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:03.148762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:03.359395 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:03.359362    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:03.693174 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:03.693094    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:04.201770 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:04.201715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:07:04.204027 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:04.204007    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:07:04.302410 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:04.302370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:04.302573 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:07:04.302537    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:07:04.302625 ip-10-0-134-83 kubenswrapper[2577]: E1031 03:07:04.302603    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls podName:fdc104e8-81dd-4bef-8dee-e28bf5c7a10d nodeName:}" failed. No retries permitted until 2026-10-31 03:07:20.302587269 +0000 UTC m=+63.789626913 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8z98n" (UID: "fdc104e8-81dd-4bef-8dee-e28bf5c7a10d") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:07:04.326928 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:04.326892    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-whz62"
Oct 31 03:07:04.487396 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:04.487368    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:04.687698 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:04.687669    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-whz62"]
Oct 31 03:07:04.691220 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:07:04.691187    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569a2ae0_d9c0_4f18_a27c_14167c5ccdcb.slice/crio-7cacd9fe2ab54e5a06edd5fd7347a283dc0f3a68502f21d14fa868da3a228bdc WatchSource:0}: Error finding container 7cacd9fe2ab54e5a06edd5fd7347a283dc0f3a68502f21d14fa868da3a228bdc: Status 404 returned error can't find the container with id 7cacd9fe2ab54e5a06edd5fd7347a283dc0f3a68502f21d14fa868da3a228bdc
Oct 31 03:07:04.891382 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:04.891293    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:05.122072 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:05.122041    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:05.308258 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:05.308226    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-whz62" event={"ID":"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb","Type":"ContainerStarted","Data":"7cacd9fe2ab54e5a06edd5fd7347a283dc0f3a68502f21d14fa868da3a228bdc"}
Oct 31 03:07:05.355781 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:05.355753    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:05.696526 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:05.696456    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:06.493003 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:06.492972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:06.885140 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:06.885055    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:07.087916 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.087881    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:07.340923 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.340889    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:07.606614 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.606540    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:07.828353 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.828315    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-ccdffd8bb-vmv6b"]
Oct 31 03:07:07.830345 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.830327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:07.882146 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.882062    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 03:07:07.882146 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.882079    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 03:07:07.902556 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.902504    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:07.931759 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.931727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c48f7097-d8ba-4a86-9fe7-d56739a332f3-image-registry-private-configuration\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:07.931942 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.931765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c48f7097-d8ba-4a86-9fe7-d56739a332f3-ca-trust-extracted\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:07.931942 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.931900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjjdc\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-kube-api-access-zjjdc\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:07.931942 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.931937    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c48f7097-d8ba-4a86-9fe7-d56739a332f3-registry-certificates\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:07.932103 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.932031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-bound-sa-token\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:07.932103 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.932071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c48f7097-d8ba-4a86-9fe7-d56739a332f3-trusted-ca\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:07.932207 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.932104    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c48f7097-d8ba-4a86-9fe7-d56739a332f3-installation-pull-secrets\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:07.932207 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:07.932131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-registry-tls\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.020129 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.020087    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 03:07:08.020608 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.020587    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xtr75\""
Oct 31 03:07:08.020838 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.020816    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 03:07:08.021270 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.021253    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ccdffd8bb-vmv6b"]
Oct 31 03:07:08.032972 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.032946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjjdc\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-kube-api-access-zjjdc\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.033095 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.032977    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c48f7097-d8ba-4a86-9fe7-d56739a332f3-registry-certificates\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.033095 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.033006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-bound-sa-token\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.033095 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.033028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c48f7097-d8ba-4a86-9fe7-d56739a332f3-trusted-ca\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.033095 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.033045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c48f7097-d8ba-4a86-9fe7-d56739a332f3-installation-pull-secrets\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.033273 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.033161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-registry-tls\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.033273 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.033187    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c48f7097-d8ba-4a86-9fe7-d56739a332f3-image-registry-private-configuration\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.033273 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.033203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c48f7097-d8ba-4a86-9fe7-d56739a332f3-ca-trust-extracted\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.033657 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.033622    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c48f7097-d8ba-4a86-9fe7-d56739a332f3-ca-trust-extracted\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.034063 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.034039    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c48f7097-d8ba-4a86-9fe7-d56739a332f3-registry-certificates\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.034161 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.034064    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c48f7097-d8ba-4a86-9fe7-d56739a332f3-trusted-ca\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.035597 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.035571    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c48f7097-d8ba-4a86-9fe7-d56739a332f3-image-registry-private-configuration\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.035675 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.035615    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c48f7097-d8ba-4a86-9fe7-d56739a332f3-installation-pull-secrets\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.035734 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.035717    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-registry-tls\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.099495 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.099463    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-bound-sa-token\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.099652 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.099552    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjjdc\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-kube-api-access-zjjdc\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.140216 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.140139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:08.269126 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.269094    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:08.561569 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.561536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:08.604225 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.604194    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ccdffd8bb-vmv6b"]
Oct 31 03:07:08.606762 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:07:08.606730    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48f7097_d8ba_4a86_9fe7_d56739a332f3.slice/crio-1f4e4ae7383bb270f7f543a25a3d55244964cf83e1ff541f37b644e7bed45f0d WatchSource:0}: Error finding container 1f4e4ae7383bb270f7f543a25a3d55244964cf83e1ff541f37b644e7bed45f0d: Status 404 returned error can't find the container with id 1f4e4ae7383bb270f7f543a25a3d55244964cf83e1ff541f37b644e7bed45f0d
Oct 31 03:07:08.936824 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:08.936734    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:09.158685 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:09.158641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:09.320404 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:09.320372    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-whz62" event={"ID":"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb","Type":"ContainerStarted","Data":"dd8ff2da1f15c0d69145a393231d2a7f85ca4123c150c6bb61cadbbb60c0c5ca"}
Oct 31 03:07:09.320554 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:09.320428    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-whz62" event={"ID":"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb","Type":"ContainerStarted","Data":"d9252d584b963ecad5ab3bcbac90b5afeae2a8dab5ccd033273c3f738d939520"}
Oct 31 03:07:09.320554 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:09.320477    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-whz62"
Oct 31 03:07:09.321847 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:09.321824    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b" event={"ID":"c48f7097-d8ba-4a86-9fe7-d56739a332f3","Type":"ContainerStarted","Data":"8efc2edbfa9839c35cad04f03edeac25c44539818108565099cb4363d899801b"}
Oct 31 03:07:09.321953 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:09.321857    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b" event={"ID":"c48f7097-d8ba-4a86-9fe7-d56739a332f3","Type":"ContainerStarted","Data":"1f4e4ae7383bb270f7f543a25a3d55244964cf83e1ff541f37b644e7bed45f0d"}
Oct 31 03:07:09.321953 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:09.321937    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:09.367613 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:09.367553    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:09.428498 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:09.428448    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-whz62" podStartSLOduration=17.557578845 podStartE2EDuration="21.428433152s" totalImagesPullingTime="3.870854307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:09.42771927 +0000 UTC m=+52.914758938" watchObservedRunningTime="2026-10-31 03:07:09.428433152 +0000 UTC m=+52.915472818"
Oct 31 03:07:09.539346 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:09.539293    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b" podStartSLOduration=2.539277912 podStartE2EDuration="2.539277912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:09.538681863 +0000 UTC m=+53.025721520" watchObservedRunningTime="2026-10-31 03:07:09.539277912 +0000 UTC m=+53.026317577"
Oct 31 03:07:09.696633 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:09.696509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:10.500530 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:10.500488    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:10.884537 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:10.884413    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:11.084302 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:11.084261    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:11.315470 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:11.315422    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:11.537835 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:11.537803    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:13.255089 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:13.255063    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:13.657256 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:13.657175    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:14.027945 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:14.027912    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:14.242168 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:14.242131    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:15.032262 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:15.032228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:15.241451 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:15.241419    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:15.442940 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:15.442861    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:15.653465 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:15.653435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:16.439041 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:16.439012    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:16.703368 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:16.703292    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:16.918880 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:16.918848    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:17.847147 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:17.847105    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:18.228992 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:18.228900    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:18.448269 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:18.448233    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:19.328119 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:19.328084    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-whz62"
Oct 31 03:07:19.381886 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:19.381856    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:19.603877 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:19.603802    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:20.309567 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:20.309538    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:20.329901 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:20.329869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:20.332270 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:20.332247    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:20.516450 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:20.516418    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:20.582361 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:20.582278    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sc79n\""
Oct 31 03:07:20.585437 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:20.585414    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:20.729978 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:20.729939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:20.946611 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:20.946526    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8z98n"]
Oct 31 03:07:20.981658 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:07:20.981632    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc104e8_81dd_4bef_8dee_e28bf5c7a10d.slice/crio-a3ad5fa672b2a2db023d36b971a7bf40bad4946facdfab4d6eabb31ff4bf5eba WatchSource:0}: Error finding container a3ad5fa672b2a2db023d36b971a7bf40bad4946facdfab4d6eabb31ff4bf5eba: Status 404 returned error can't find the container with id a3ad5fa672b2a2db023d36b971a7bf40bad4946facdfab4d6eabb31ff4bf5eba
Oct 31 03:07:21.363007 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:21.362974    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8z98n" event={"ID":"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d","Type":"ContainerStarted","Data":"f8b6b8c5e3322f96fe7f024b5aef24dc69a1d77a98a71f8e2e672d53dbfd17f0"}
Oct 31 03:07:21.363393 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:21.363032    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8z98n" event={"ID":"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d","Type":"ContainerStarted","Data":"a3ad5fa672b2a2db023d36b971a7bf40bad4946facdfab4d6eabb31ff4bf5eba"}
Oct 31 03:07:22.128505 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:22.128472    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:22.333877 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:22.333842    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:22.947704 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:22.947661    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:07:22.950462 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:22.950438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:07:23.166900 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:23.166809    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ffsgr\""
Oct 31 03:07:23.169854 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:23.169836    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:07:23.371265 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:23.371229    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8z98n" event={"ID":"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d","Type":"ContainerStarted","Data":"750b32c56df7dc4d8427f02165265b945ffa78679ef8089d1dcea4a6c9d1271a"}
Oct 31 03:07:23.529209 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:23.529178    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-58h48"]
Oct 31 03:07:23.532057 ip-10-0-134-83 kubenswrapper[2577]: W1031 03:07:23.532030    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bcc2a31_3406_4199_b08d_1ad8817a5cc2.slice/crio-a43e9a82fbbc81b5c0e127f3f8ed690d7fa159485006e14ff5b543613089062d WatchSource:0}: Error finding container a43e9a82fbbc81b5c0e127f3f8ed690d7fa159485006e14ff5b543613089062d: Status 404 returned error can't find the container with id a43e9a82fbbc81b5c0e127f3f8ed690d7fa159485006e14ff5b543613089062d
Oct 31 03:07:23.858201 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:23.858098    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:24.071138 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:24.071097    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:24.274938 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:24.274903    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:24.377058 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:24.377014    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-58h48" event={"ID":"9bcc2a31-3406-4199-b08d-1ad8817a5cc2","Type":"ContainerStarted","Data":"a43e9a82fbbc81b5c0e127f3f8ed690d7fa159485006e14ff5b543613089062d"}
Oct 31 03:07:24.694543 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:24.694342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:24.920490 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:24.920447    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:25.208725 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:25.208692    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:25.382968 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:25.382931    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-58h48" event={"ID":"9bcc2a31-3406-4199-b08d-1ad8817a5cc2","Type":"ContainerStarted","Data":"0ac8d505c897a922051787ba4d9fc779445dd70fa170e83248b1d0aadb310aa0"}
Oct 31 03:07:25.383121 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:25.382996    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-58h48" event={"ID":"9bcc2a31-3406-4199-b08d-1ad8817a5cc2","Type":"ContainerStarted","Data":"1ea73548b30beb295902f71bd2a7718c4db5ab3730b5def00e6ef766d6ee0c07"}
Oct 31 03:07:25.492422 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:25.492361    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-58h48" podStartSLOduration=65.851146699 podStartE2EDuration="1m7.492341117s" totalImagesPullingTime="1.641194418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:25.49094547 +0000 UTC m=+68.977985136" watchObservedRunningTime="2026-10-31 03:07:25.492341117 +0000 UTC m=+68.979380784"
Oct 31 03:07:25.649771 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:25.649726    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:25.861463 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:25.861366    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:26.641752 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:26.641715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:26.852372 ip-10-0-134-83 kubenswrapper[2577]: I1031 03:07:26.852332    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:26.972773 ip-10-0-134-83 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 03:07:26.987219 ip-10-0-134-83 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 03:07:26.987390 ip-10-0-134-83 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 03:07:26.987448 ip-10-0-134-83 systemd[1]: kubelet.service: Consumed 3.317s CPU time.
Oct 31 03:07:27.001341 ip-10-0-134-83 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 03:07:27.062192 ip-10-0-134-83 kubenswrapper[4900]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:07:27.062192 ip-10-0-134-83 kubenswrapper[4900]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 03:07:27.062192 ip-10-0-134-83 kubenswrapper[4900]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:07:27.062192 ip-10-0-134-83 kubenswrapper[4900]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:07:27.064485 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064403    4900 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:27.064485 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064486    4900 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:27.064485 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064490    4900 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064493    4900 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064496    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064499    4900 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064503    4900 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064514    4900 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064519    4900 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064522    4900 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064525    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064528    4900 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064532    4900 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064535    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064538    4900 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064541    4900 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064544    4900 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064546    4900 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064549    4900 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064552    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064554    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:27.064579 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064557    4900 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064560    4900 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064563    4900 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064565    4900 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064568    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064571    4900 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064573    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064577    4900 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064580    4900 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064582    4900 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064585    4900 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064588    4900 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064592    4900 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064595    4900 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064598    4900 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064600    4900 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064603    4900 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064605    4900 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064610    4900 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064614    4900 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:27.065075 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064617    4900 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064620    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064623    4900 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064626    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064628    4900 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064631    4900 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064634    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064636    4900 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064639    4900 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064642    4900 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064645    4900 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064648    4900 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064650    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064654    4900 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064657    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064659    4900 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064705    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064709    4900 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064712    4900 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064716    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:27.065557 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064719    4900 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064722    4900 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064724    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064727    4900 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064730    4900 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064733    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064736    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064739    4900 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064742    4900 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064764    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064769    4900 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064773    4900 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064776    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064778    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064781    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064798    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064802    4900 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064805    4900 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064807    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:27.066173 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064810    4900 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064813    4900 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064816    4900 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064818    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064821    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064824    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064827    4900 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064830    4900 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064832    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064835    4900 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064838    4900 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064841    4900 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064844    4900 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064846    4900 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064849    4900 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064852    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064855    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064858    4900 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064860    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:27.066634 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064863    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064865    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064868    4900 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064872    4900 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064877    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064879    4900 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.064883    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065762    4900 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065769    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065775    4900 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065778    4900 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065781    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065800    4900 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065805    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065809    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065812    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065817    4900 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065821    4900 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065824    4900 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:27.067132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065828    4900 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065830    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065833    4900 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065836    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065839    4900 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065842    4900 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065845    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065847    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065850    4900 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065853    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065855    4900 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065858    4900 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065861    4900 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065863    4900 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065866    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065868    4900 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065871    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065875    4900 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065878    4900 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:27.067677 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065880    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065884    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065889    4900 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065893    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065896    4900 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065899    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065902    4900 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065905    4900 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065908    4900 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065910    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065913    4900 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065916    4900 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065919    4900 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065921    4900 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065924    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065927    4900 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065930    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065932    4900 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065935    4900 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:27.068158 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065938    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065940    4900 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065943    4900 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065945    4900 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065948    4900 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065951    4900 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065953    4900 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065956    4900 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065959    4900 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065962    4900 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065965    4900 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065968    4900 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065971    4900 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065973    4900 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065976    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065979    4900 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065982    4900 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065985    4900 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065987    4900 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065990    4900 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:27.068673 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065993    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065996    4900 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.065998    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066001    4900 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066004    4900 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066006    4900 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066009    4900 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066011    4900 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066014    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066017    4900 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066019    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066022    4900 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066024    4900 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066027    4900 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066030    4900 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066033    4900 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066035    4900 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066038    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066040    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:27.069196 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066043    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066046    4900 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066048    4900 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066051    4900 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066054    4900 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066057    4900 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066060    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066063    4900 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066080    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066091    4900 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066094    4900 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066098    4900 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066101    4900 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066104    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066106    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066110    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066113    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066603    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066609    4900 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066613    4900 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:27.069655 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066616    4900 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066619    4900 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066622    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066625    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066628    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066631    4900 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066634    4900 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066636    4900 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066640    4900 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066643    4900 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066646    4900 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066649    4900 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066652    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066654    4900 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066658    4900 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066660    4900 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066663    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066666    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066669    4900 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066671    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:27.070183 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066674    4900 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066677    4900 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066680    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066683    4900 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066686    4900 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066690    4900 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066693    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066696    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066699    4900 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066702    4900 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066705    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066710    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066713    4900 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066716    4900 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066718    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066721    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066724    4900 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066726    4900 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066730    4900 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:27.070674 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066732    4900 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066736    4900 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066738    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066741    4900 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066744    4900 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066746    4900 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066749    4900 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066752    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066755    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066758    4900 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066760    4900 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066763    4900 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066766    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066768    4900 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066771    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066774    4900 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066777    4900 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066780    4900 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066798    4900 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:27.071144 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066803    4900 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066806    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066810    4900 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066813    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066816    4900 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066820    4900 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066822    4900 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066825    4900 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066828    4900 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066830    4900 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066833    4900 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066836    4900 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066838    4900 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066841    4900 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066844    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066847    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066850    4900 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066852    4900 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066855    4900 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066858    4900 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:27.071989 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066860    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066863    4900 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066866    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066868    4900 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066871    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066873    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066876    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066879    4900 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066881    4900 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066884    4900 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066887    4900 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066890    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066892    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066895    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066898    4900 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066901    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066903    4900 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066912    4900 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066916    4900 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:27.072698 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066919    4900 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066922    4900 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066925    4900 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066928    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066932    4900 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.066935    4900 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067034    4900 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067041    4900 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067048    4900 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067053    4900 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067057    4900 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067060    4900 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067067    4900 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067072    4900 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067075    4900 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067078    4900 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067081    4900 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067085    4900 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067088    4900 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067091    4900 flags.go:64] FLAG: --cgroup-root=""
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067094    4900 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067097    4900 flags.go:64] FLAG: --client-ca-file=""
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067100    4900 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067104    4900 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 03:07:27.073184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067108    4900 flags.go:64] FLAG: --cluster-domain=""
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067111    4900 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067114    4900 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067117    4900 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067121    4900 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067125    4900 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067130    4900 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067133    4900 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067136    4900 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067140    4900 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067144    4900 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067147    4900 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067150    4900 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067153    4900 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067157    4900 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067161    4900 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067164    4900 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067167    4900 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067170    4900 flags.go:64] FLAG: --enable-server="true"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067173    4900 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067178    4900 flags.go:64] FLAG: --event-burst="100"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067181    4900 flags.go:64] FLAG: --event-qps="50"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067184    4900 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067188    4900 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 03:07:27.073751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067191    4900 flags.go:64] FLAG: --eviction-hard=""
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067195    4900 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067198    4900 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067201    4900 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067205    4900 flags.go:64] FLAG: --eviction-soft=""
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067208    4900 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067211    4900 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067214    4900 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067218    4900 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067221    4900 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067224    4900 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067227    4900 flags.go:64] FLAG: --feature-gates=""
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067231    4900 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067234    4900 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067239    4900 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067242    4900 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067245    4900 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067250    4900 flags.go:64] FLAG: --help="false"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067253    4900 flags.go:64] FLAG: --hostname-override="ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067257    4900 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067260    4900 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067263    4900 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067267    4900 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 03:07:27.074394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067270    4900 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067273    4900 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067276    4900 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067279    4900 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067282    4900 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067285    4900 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067289    4900 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067292    4900 flags.go:64] FLAG: --kube-reserved=""
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067295    4900 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067298    4900 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067301    4900 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067304    4900 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067307    4900 flags.go:64] FLAG: --lock-file=""
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067310    4900 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067313    4900 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067316    4900 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067323    4900 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067326    4900 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067329    4900 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067333    4900 flags.go:64] FLAG: --logging-format="text"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067336    4900 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067339    4900 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067342    4900 flags.go:64] FLAG: --manifest-url=""
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067347    4900 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067352    4900 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 03:07:27.074976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067355    4900 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067360    4900 flags.go:64] FLAG: --max-pods="110"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067364    4900 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067367    4900 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067370    4900 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067373    4900 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067376    4900 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067379    4900 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067382    4900 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067389    4900 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067392    4900 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067396    4900 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067399    4900 flags.go:64] FLAG: --pod-cidr=""
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067402    4900 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067405    4900 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067408    4900 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067411    4900 flags.go:64] FLAG: --port="10250"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067414    4900 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067417    4900 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-06d809d645533215d"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067421    4900 flags.go:64] FLAG: --qos-reserved=""
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067424    4900 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067427    4900 flags.go:64] FLAG: --register-node="true"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067430    4900 flags.go:64] FLAG: --register-with-taints=""
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067434    4900 flags.go:64] FLAG: --registry-burst="10"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067437    4900 flags.go:64] FLAG: --registry-qps="5"
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067440    4900 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 03:07:27.075591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067443    4900 flags.go:64] FLAG: --reserved-memory=""
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067447    4900 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067450    4900 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067453    4900 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067458    4900 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067461    4900 flags.go:64] FLAG: --runonce="false"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067464    4900 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067469    4900 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067472    4900 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067475    4900 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067478    4900 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067482    4900 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067485    4900 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067488    4900 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067492    4900 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067495    4900 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067498    4900 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067501    4900 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067504    4900 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067507    4900 flags.go:64] FLAG: --system-cgroups=""
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067510    4900 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067515    4900 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067518    4900 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067521    4900 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067525    4900 flags.go:64] FLAG: --tls-min-version=""
Oct 31 03:07:27.076233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067528    4900 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067531    4900 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067534    4900 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067537    4900 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067540    4900 flags.go:64] FLAG: --v="2"
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067544    4900 flags.go:64] FLAG: --version="false"
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067548    4900 flags.go:64] FLAG: --vmodule=""
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067552    4900 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.067556    4900 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067689    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067693    4900 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067697    4900 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067702    4900 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067706    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067710    4900 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067713    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067716    4900 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067719    4900 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067721    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067724    4900 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067727    4900 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:27.076859 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067730    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067733    4900 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067736    4900 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067738    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067741    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067744    4900 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067747    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067750    4900 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067752    4900 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067755    4900 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067757    4900 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067760    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067763    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067765    4900 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067768    4900 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067771    4900 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067774    4900 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067776    4900 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067779    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067782    4900 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:27.077368 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067799    4900 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067803    4900 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067807    4900 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067810    4900 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067812    4900 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067816    4900 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067819    4900 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067822    4900 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067825    4900 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067827    4900 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067830    4900 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067833    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067835    4900 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067838    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067841    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067843    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067846    4900 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067849    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067852    4900 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067855    4900 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:27.077851 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067857    4900 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067860    4900 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067863    4900 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067865    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067868    4900 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067871    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067873    4900 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067876    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067879    4900 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067881    4900 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067884    4900 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067886    4900 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067889    4900 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067892    4900 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067896    4900 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067899    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067902    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067906    4900 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067908    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067911    4900 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:27.078386 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067914    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067917    4900 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067920    4900 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067924    4900 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067928    4900 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067932    4900 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067935    4900 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067939    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067942    4900 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067945    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067948    4900 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067950    4900 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067953    4900 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067956    4900 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067958    4900 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067961    4900 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067964    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067967    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067969    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:27.079009 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067972    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067975    4900 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067977    4900 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067980    4900 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067983    4900 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067985    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067988    4900 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067993    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067996    4900 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.067999    4900 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068003    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068006    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068009    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068011    4900 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068014    4900 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.068040    4900 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:07:27.079724 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068077    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068081    4900 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068084    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068087    4900 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068090    4900 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068093    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068095    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068098    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068101    4900 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068104    4900 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068106    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068109    4900 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068112    4900 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068116    4900 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068120    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068123    4900 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068125    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068128    4900 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068131    4900 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:27.080490 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068134    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068136    4900 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068139    4900 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068142    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068147    4900 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068150    4900 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068153    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068155    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068160    4900 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068163    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068165    4900 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068168    4900 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068171    4900 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068174    4900 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068176    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068179    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068182    4900 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068185    4900 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068188    4900 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:27.081148 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068190    4900 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068193    4900 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068196    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068198    4900 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068201    4900 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068204    4900 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068207    4900 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068209    4900 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068212    4900 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068215    4900 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068217    4900 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068220    4900 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068222    4900 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068225    4900 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068227    4900 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068230    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068233    4900 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068236    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068239    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068242    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:27.081756 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068245    4900 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068249    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068252    4900 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068255    4900 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068257    4900 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068260    4900 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068263    4900 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068265    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068268    4900 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068271    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068274    4900 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068276    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068279    4900 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068281    4900 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068284    4900 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068287    4900 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068290    4900 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068292    4900 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068295    4900 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068298    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:27.082279 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068300    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068304    4900 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068306    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068309    4900 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068312    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068314    4900 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068317    4900 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068321    4900 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068324    4900 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068329    4900 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068331    4900 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068335    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068338    4900 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068342    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068345    4900 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068347    4900 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068350    4900 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068353    4900 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068355    4900 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:27.082819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068358    4900 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068361    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068363    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068366    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068369    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068371    4900 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068374    4900 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068377    4900 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.068380    4900 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.073452    4900 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.073485    4900 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.073506    4900 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.073511    4900 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073570    4900 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073576    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073579    4900 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073582    4900 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073585    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073588    4900 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:27.083289 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073591    4900 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073593    4900 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073596    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073599    4900 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073602    4900 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073605    4900 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073608    4900 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073610    4900 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073614    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073617    4900 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073620    4900 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073623    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073626    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073628    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073631    4900 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073634    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073637    4900 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073639    4900 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073642    4900 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:27.083768 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073645    4900 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073648    4900 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073651    4900 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073654    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073656    4900 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073659    4900 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073663    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073665    4900 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073668    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073671    4900 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073674    4900 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073676    4900 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073679    4900 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073682    4900 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073684    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073687    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073690    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073693    4900 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073696    4900 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:27.084398 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073699    4900 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073702    4900 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073705    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073709    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073711    4900 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073714    4900 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073717    4900 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073720    4900 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073723    4900 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073726    4900 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073730    4900 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073735    4900 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073739    4900 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073743    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073746    4900 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073749    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073753    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073756    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073759    4900 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:27.084883 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073762    4900 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073765    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073767    4900 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073770    4900 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073773    4900 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073776    4900 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073778    4900 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073781    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073797    4900 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073802    4900 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073806    4900 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073810    4900 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073812    4900 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073815    4900 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073818    4900 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073820    4900 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073825    4900 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073828    4900 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073830    4900 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073833    4900 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:27.085350 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073836    4900 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073839    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073841    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073844    4900 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073846    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073849    4900 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073852    4900 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073855    4900 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073858    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073861    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073863    4900 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073866    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073869    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073872    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073875    4900 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073877    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073880    4900 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073883    4900 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073885    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:27.085844 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073888    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073891    4900 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073894    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.073896    4900 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.073926    4900 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074070    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074075    4900 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074079    4900 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074082    4900 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074085    4900 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074088    4900 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074091    4900 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074094    4900 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074097    4900 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074100    4900 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074103    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:27.086306 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074105    4900 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074108    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074111    4900 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074114    4900 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074117    4900 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074119    4900 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074122    4900 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074125    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074128    4900 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074130    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074133    4900 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074136    4900 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074139    4900 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074141    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074144    4900 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074147    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074150    4900 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074152    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074155    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074158    4900 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:27.086690 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074160    4900 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074163    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074167    4900 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074171    4900 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074174    4900 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074178    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074181    4900 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074184    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074187    4900 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074190    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074193    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074196    4900 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074199    4900 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074201    4900 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074204    4900 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074207    4900 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074209    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074212    4900 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074215    4900 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:27.087181 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074217    4900 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074220    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074223    4900 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074225    4900 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074228    4900 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074231    4900 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074234    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074236    4900 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074239    4900 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074242    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074245    4900 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074247    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074250    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074253    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074255    4900 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074258    4900 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074260    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074263    4900 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074266    4900 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:27.087809 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074268    4900 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074271    4900 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074274    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074277    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074280    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074283    4900 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074286    4900 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074289    4900 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074291    4900 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074294    4900 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074297    4900 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074299    4900 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074302    4900 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074305    4900 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074308    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074311    4900 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074314    4900 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074317    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074319    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:27.088535 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074322    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074324    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074327    4900 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074330    4900 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074333    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074335    4900 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074338    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074340    4900 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074343    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074346    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074349    4900 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074351    4900 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074354    4900 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074357    4900 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074359    4900 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074362    4900 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074366    4900 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:27.089104 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074370    4900 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.074393    4900 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074426    4900 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074432    4900 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074436    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074442    4900 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074448    4900 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074454    4900 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074459    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074463    4900 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074468    4900 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074474    4900 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074479    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074482    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074485    4900 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074489    4900 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:07:27.089522 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074491    4900 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074494    4900 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074497    4900 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074500    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074502    4900 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074505    4900 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074508    4900 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074511    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074513    4900 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074516    4900 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074519    4900 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074522    4900 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074524    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074527    4900 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074530    4900 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074533    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074536    4900 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074539    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074542    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:07:27.090032 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074544    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074547    4900 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074550    4900 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074552    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074555    4900 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074558    4900 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074560    4900 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074563    4900 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074566    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074568    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074571    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074574    4900 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074576    4900 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074579    4900 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074587    4900 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074590    4900 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074592    4900 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074595    4900 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074598    4900 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:07:27.090484 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074601    4900 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074603    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074606    4900 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074609    4900 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074611    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074614    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074617    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074620    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074623    4900 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074625    4900 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074628    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074632    4900 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074635    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074637    4900 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074640    4900 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074643    4900 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074645    4900 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074648    4900 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074651    4900 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074653    4900 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:07:27.090950 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074656    4900 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074659    4900 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074663    4900 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074667    4900 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074670    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074673    4900 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074675    4900 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074680    4900 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074683    4900 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074686    4900 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074688    4900 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074691    4900 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074694    4900 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074697    4900 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074700    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074704    4900 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074707    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074710    4900 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074712    4900 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074715    4900 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:07:27.091420 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074718    4900 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074721    4900 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074723    4900 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074726    4900 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074729    4900 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074732    4900 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074735    4900 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074738    4900 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074740    4900 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074743    4900 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074746    4900 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074749    4900 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074751    4900 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:27.074754    4900 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.074837    4900 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.075839    4900 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.075896    4900 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.076073    4900 server.go:1014] "Starting client certificate rotation"
Oct 31 03:07:27.091963 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.076207    4900 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:07:27.092471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.076258    4900 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:01:17 +0000 UTC" deadline="2028-04-13 13:23:50.600677798 +0000 UTC"
Oct 31 03:07:27.092471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.076286    4900 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12730h16m23.524393816s"
Oct 31 03:07:27.092471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.076513    4900 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:07:27.092471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.077621    4900 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:07:27.092471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.078861    4900 log.go:25] "Validated CRI v1 runtime API"
Oct 31 03:07:27.092471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.080808    4900 log.go:25] "Validated CRI v1 image API"
Oct 31 03:07:27.092471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.081463    4900 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 03:07:27.092471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.083116    4900 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b81b264a-72e5-490a-836a-1722ad0dc5bc:/dev/nvme0n1p4 d0dd9871-4a01-48fe-ad50-170d5903f4c7:/dev/nvme0n1p3]
Oct 31 03:07:27.092694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.083136    4900 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1f4e4ae7383bb270f7f543a25a3d55244964cf83e1ff541f37b644e7bed45f0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1f4e4ae7383bb270f7f543a25a3d55244964cf83e1ff541f37b644e7bed45f0d/userdata/shm major:0 minor:251 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d3fe1d96745eb4e733fc9f89eebf2eda108501e387f4349ffb9579ffdfa4c56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d3fe1d96745eb4e733fc9f89eebf2eda108501e387f4349ffb9579ffdfa4c56/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45f57db50ba1789cd6bb63193217cd822742e83154a9cfeb674d13ce7e0b0056/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45f57db50ba1789cd6bb63193217cd822742e83154a9cfeb674d13ce7e0b0056/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ec5f69d636b214266e4afa421e5792d890a81db8f8e8f5db5cad39d0e21e8c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ec5f69d636b214266e4afa421e5792d890a81db8f8e8f5db5cad39d0e21e8c4/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5dd6113daff0a3d568c8b28c559270b7619b13e6d7490bbad4a71a3e5c3d9183/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5dd6113daff0a3d568c8b28c559270b7619b13e6d7490bbad4a71a3e5c3d9183/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/622aafbf0dd8f040baf632757fa42b94b6b3f2b35c29f708587332a38c5e5f82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/622aafbf0dd8f040baf632757fa42b94b6b3f2b35c29f708587332a38c5e5f82/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/64491765ef85d4cabbae65ed27e83cbb6188588cf0895952d26d16d7f6071fed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/64491765ef85d4cabbae65ed27e83cbb6188588cf0895952d26d16d7f6071fed/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/709a3960720e1f59c1ae8a7a1f387366ff04f69fa827dec3ba5f6673185ab7c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/709a3960720e1f59c1ae8a7a1f387366ff04f69fa827dec3ba5f6673185ab7c8/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7cacd9fe2ab54e5a06edd5fd7347a283dc0f3a68502f21d14fa868da3a228bdc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7cacd9fe2ab54e5a06edd5fd7347a283dc0f3a68502f21d14fa868da3a228bdc/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/876a361fa14fd14406a22cd3e2e4150563d7a08a7131204279c32ab1bb2fc1fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/876a361fa14fd14406a22cd3e2e4150563d7a08a7131204279c32ab1bb2fc1fc/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8eeb180f70ec034f4e3eea020d257929f75b83377a99352b01db8234a02d3ca7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8eeb180f70ec034f4e3eea020d257929f75b83377a99352b01db8234a02d3ca7/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b516d30a0d2610ab2354787c4667cf4555c8a1ce3ef2c105dff044a87d227c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b516d30a0d2610ab2354787c4667cf4555c8a1ce3ef2c105dff044a87d227c6/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a3ad5fa672b2a2db023d36b971a7bf40bad4946facdfab4d6eabb31ff4bf5eba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a3ad5fa672b2a2db023d36b971a7bf40bad4946facdfab4d6eabb31ff4bf5eba/userdata/shm major:0 minor:273 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a43e9a82fbbc81b5c0e127f3f8ed690d7fa159485006e14ff5b543613089062d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a43e9a82fbbc81b5c0e127f3f8ed690d7fa159485006e14ff5b543613089062d/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab67e784a14ff154144d603e90c8fce7bff0153a910b8f931121c80a8eba8ab9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab67e784a14ff154144d603e90c8fce7bff0153a910b8f931121c80a8eba8ab9/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b13e823829260663bb49df1ca87e7d3309632c077cbd5b794617aaadedfd598b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b13e823829260663bb49df1ca87e7d3309632c077cbd5b794617aaadedfd598b/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/caa2b56f9caef11de2206811cb5d6524aba33a4d8e287fd1f6583f45e991c4fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/caa2b56f9caef11de2206811cb5d6524aba33a4d8e287fd1f6583f45e991c4fd/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e49095abaa2842e614824b2264069e2370b832b0454c1cedf7577eb10be3f11b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e49095abaa2842e614824b2264069e2370b832b0454c1cedf7577eb10be3f11b/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/056de880-e0a6-4b20-805f-669a017b3d1d/volumes/kubernetes.io~projected/kube-api-access-shnrb:{mountpoint:/var/lib/kubelet/pods/056de880-e0a6-4b20-805f-669a017b3d1d/volumes/kubernetes.io~projected/kube-api-access-shnrb major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cb0ab42-11a8-493c-8218-4755fb32e543/volumes/kubernetes.io~projected/kube-api-access-rhdgp:{mountpoint:/var/lib/kubelet/pods/2cb0ab42-11a8-493c-8218-4755fb32e543/volumes/kubernetes.io~projected/kube-api-access-rhdgp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3313df2c-6c0b-44d8-a356-8da87e1d620a/volumes/kubernetes.io~projected/kube-api-access-fptbl:{mountpoint:/var/lib/kubelet/pods/3313df2c-6c0b-44d8-a356-8da87e1d620a/volumes/kubernetes.io~projected/kube-api-access-fptbl major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3313df2c-6c0b-44d8-a356-8da87e1d620a/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/3313df2c-6c0b-44d8-a356-8da87e1d620a/volumes/kubernetes.io~secret/signing-key major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47d4b722-8199-4e8e-87bf-915e1ac409f5/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/47d4b722-8199-4e8e-87bf-915e1ac409f5/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47d4b722-8199-4e8e-87bf-915e1ac409f5/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/47d4b722-8199-4e8e-87bf-915e1ac409f5/volumes/kubernetes.io~empty-dir/tmp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/47d4b722-8199-4e8e-87bf-915e1ac409f5/volumes/kubernetes.io~projected/kube-api-access-lzswc:{mountpoint:/var/lib/kubelet/pods/47d4b722-8199-4e8e-87bf-915e1ac409f5/volumes/kubernetes.io~projected/kube-api-access-lzswc major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52b495cc-bf15-4438-8762-aab455e12701/volumes/kubernetes.io~projected/kube-api-access-lwdf9:{mountpoint:/var/lib/kubelet/pods/52b495cc-bf15-4438-8762-aab455e12701/volumes/kubernetes.io~projected/kube-api-access-lwdf9 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb/volumes/kubernetes.io~projected/kube-api-access-657qr:{mountpoint:/var/lib/kubelet/pods/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb/volumes/kubernetes.io~projected/kube-api-access-657qr major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb/volumes/kubernetes.io~secret/metrics-tls major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b5e3f9c-61e6-4468-9891-afdeb408caf7/volumes/kubernetes.io~projected/kube-api-access-ltgxk:{mountpoint:/var/lib/kubelet/pods/8b5e3f9c-61e6-4468-9891-afdeb408caf7/volumes/kubernetes.io~projected/kube-api-access-ltgxk major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b5e3f9c-61e6-4468-9891-afdeb408caf7/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8b5e3f9c-61e6-4468-9891-afdeb408caf7/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90b722fb-cc4e-4c62-877b-db0df5c5d16c/volumes/kubernetes.io~projected/kube-api-access-xlbbg:{mountpoint:/var/lib/kubelet/pods/90b722fb-cc4e-4c62-877b-db0df5c5d16c/volumes/kubernetes.io~projected/kube-api-access-xlbbg major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bcc2a31-3406-4199-b08d-1ad8817a5cc2/volumes/kubernetes.io~projected/kube-api-access-xwwgj:{mountpoint:/var/lib/kubelet/pods/9bcc2a31-3406-4199-b08d-1ad8817a5cc2/volumes/kubernetes.io~projected/kube-api-access-xwwgj major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bcc2a31-3406-4199-b08d-1ad8817a5cc2/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/9bcc2a31-3406-4199-b08d-1ad8817a5cc2/volumes/kubernetes.io~secret/metrics-certs major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a957db89-5022-4877-aa05-3713401881ed/volumes/kubernetes.io~projected/kube-api-access-l6k56:{mountpoint:/var/lib/kubelet/pods/a957db89-5022-4877-aa05-3713401881ed/volumes/kubernetes.io~projected/kube-api-access-l6k56 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~projected/kube-api-access-zjjdc:{mountpoint:/var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~projected/kube-api-access-zjjdc major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~projected/registry-tls major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cda5dcc5-e5c6-4e0a-9acb-6191475f552c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/cda5dcc5-e5c6-4e0a-9acb-6191475f552c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cda5dcc5-e5c6-4e0a-9acb-6191475f552c/volumes/kubernetes.io~projected/kube-api-access-zcl8m:{mountpoint:/var/lib/kubelet/pods/cda5dcc5-e5c6-4e0a-9acb-6191475f552c/volumes/kubernetes.io~projected/kube-api-access-zcl8m major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cda5dcc5-e5c6-4e0a-9acb-6191475f552c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/cda5dcc5-e5c6-4e0a-9acb-6191475f552c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5c68fab-89c6-485c-a250-cb958115e5a2/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f5c68fab-89c6-485c-a250-cb958115e5a2/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f714c9b3-ccf2-4dab-9220-321c5b4258d0/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f714c9b3-ccf2-4dab-9220-321c5b4258d0/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa/volumes/kubernetes.io~projected/kube-api-access-7dbdb:{mountpoint:/var/lib/kubelet/pods/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa/volumes/kubernetes.io~projected/kube-api-access-7dbdb major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d/volumes/kubernetes.io~projected/kube-api-access-knqzf:{mountpoint:/var/lib/kubelet/pods/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d/volumes/kubernetes.io~projected/kube-api-access-knqzf major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdf1fa41-a03b-4a64-ba80-4c7a2825e259/volumes/kubernetes.io~projected/kube-api-access-lblgh:{mountpoint:/var/lib/kubelet/pods/fdf1fa41-a03b-4a64-ba80-4c7a2825e259/volumes/kubernetes.io~projected/kube-api-access-lblgh major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/900ce355aa38de145ebc74b29ce10259cae6824dc1cede94d7f918d64ef61642/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0f82ed6c4f60cdc14453155e7ac3fa931c6856a03d11ac35c59b4bc3677d6738/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0f496ffd05ab8c255a1c7760cb7d8f3889854a56fbe02043043d9d4bc914d74e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d277eb7092a3c77c91cfef2089912d57f0cd3a2a34a0f178320f6790f546eb99/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/1149127ed964a4119c3e9ebe61f49016ba39a1a0256ece32ff14bdc28c8daf16/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b85531fd614ea372bf1ae9ce4de0bfb6110685044456b52ddcdad30d7c82105f/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/08e8bbe961452d6aa8c0444887c7f1a46b4e9bcab256afc80a3788f42af8eaf8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2dd2d08bc349490a8ed33f059d996059116ee1dcf270d63a20e9c826e1dcf37f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/308783a4476013b14db16ebd518227008c751c2512ac43eef34139f7795a9f76/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/7d40640e2789da1ca54c1bfed935a457724b0517b4c42b22c0faae7e8a6a5a78/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/1aa88f58bcc13800767a86f09418ba601c206dacbcaeb18538f5fe0fde719f38/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/1b1f594b7c18ae0199a7d3ca4546926cf64da0cfb39d9e7f782f9fff60f5edc2/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/10fe94b1c8f56d11e2a21bb5a4e21c50738f6724af1e6bf7ea9206f0d6cccbe5/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/aba3f42cbaec9d179304833a0977699e72f6fe3b70a4b8c907bd30d22db5cdd1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7ba28a1c49b3b96793dd32f8d3291f8d31d0cf1a02d17b70a3b2a0912651df31/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c4ee9ef9434fa99e0509558041fbbef03f348d67b2cb44fd386d326f6b0f3fa2/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/29cf9613260e3b1b3e3a7b0c007e59085cdc9610d145a969be93f4e3caf993ea/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b1557307639dc7c71d3c7c86425ad9c3021b7e0fdc19fe572a501f967a6517dd/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/582b6dd0d89ccc5f4bc5131febc3a831014898e1fbcc3fdb6a745a3715c870f9/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/be957630f2adbb3f9508e2f8fc0fd6ca0ecba655569142e06d75356ef7290d80/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/bb118d01ad548fcbd79900c93f40b5e30d5cb683b375318a05363f86ce6a2ef3/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/4a69e90df2ac65383672528743a91157890a1c1b2a180af708cee59e4b087f2e/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/1fade7f44c489d7a390e57d2592731b416f17f89420f17d873464306eb4d243f/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/23e3732860d5e05ac635afcddc1958ff407e48b7b51b2aed5bbd74fd9562af97/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/48015f5c81fdf1204ea8d1b1835766b02ae960a370386b164ba8cfcb28d9a60f/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/8d90e969e290e6f4618cf25641d65e6aec0bf60980eb59fc1704707dbd1cb230/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/3f9b4c0a140ff62f15d12a475bddc2977732faec13c9c051ac90b627778f7b8a/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/772b94d40d0b310aca5e8de4074dd258b79ac4c0d3489c0cf2ee67ca36ac5b03/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/4069ecde30a01ac98e0fc7537c2f0efd62b8d8462b8e1f7c21c7113995fc5d98/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/c3a85fc54026dfd427c7100684528a4bb76ba2b734a2b217514bc0328eecf1c0/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/4e2a3daafbca244dfc66270cc00db44200349f88534c92f7d42c2cc0c825e489/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/3ed92c7acd9d8cf565ce2c4a149625e386ba0cc79156f7104470f172e96ab094/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/1588b0f75608c0256b35ca6ff30b691797e8cfc994a8abef312c0b38c124854b/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/6f0da6a46e9ec16091dac203491bec691c1035be0bbd03cc5aa6e9991f60fa6f/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/d7f3f7fb62dd294d9438eb6dbd83c63559f7e6ca2949d6fabf47e8898839cd97/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/d319c836e2dbd46c085f5e19cce82a3f7c78d9fe7f5b308932d19a19e248b9c4/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/9345301e7dcaa4488464614ec5ab33f75e62151a5e01df522f5d2123d38c020b/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/0b99ca76d9ceb9ba12034ce92a29d018b727fa44bdbe6042e6fe8fb8a4f94fa9/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/02b0794b887e23ffe90e43d5c38dbeea8cfa02383110ce2f062b017f543ffd18/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/651ce1fbd088fe6d5617c2721ff09f29ca35d317d08814d95026c8d14ca50b60/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/19163ab197011e62fa381da3c1825fff3fdb964bdf5a88ebf62a9480209b7893/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/19d95c0dbd842eba09f07bcc1b062af9e910c4440c75590da0e7796f6ef7824a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a4e2465c9f3529dd6b0b40f05da9d855092f32cd712ac75e0ad9f9f33dd071d3/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/2e115720facd6931a8105246c02ddc6b67507be9bd802f6f2dc93401007dcb24/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/73e16527e5af1707ff896e10c2a1e03b3897a739f7d969c981b7f5be5165b899/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/a74d8612140338b2b72b992e501f0139c9cf13cdd6eb5cfc11f73440248d6064/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/724abf2634e52b09cde1a50a5073c511a26cf49197bc0a1c8cb949875c30478f/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/b1ff5efd2e24cf7495b110b02df477f61b269bd19a473909f7761d2389765405/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8ca4bb3f964e7cd24595b101ff6bc38281d706bbf93afd27070ff6b95062653b/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4fb61c07306636001aaa1ed8322bbf00d2bea6b1f1d799341e46e1718d5a34d6/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.090146    4900 manager.go:217] Machine: {Timestamp:2026-10-31 03:07:27.089667063 +0000 UTC m=+0.056000398 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102518 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23dad2cbb57f637e48b7db7ee22a9e SystemUUID:ec23dad2-cbb5-7f63-7e48-b7db7ee22a9e BootID:c28f07c8-2d0d-48f2-809f-ddb4e5802190 Filesystems:[{Device:/var/lib/kubelet/pods/90b722fb-cc4e-4c62-877b-db0df5c5d16c/volumes/kubernetes.io~projected/kube-api-access-xlbbg DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3313df2c-6c0b-44d8-a356-8da87e1d620a/volumes/kubernetes.io~projected/kube-api-access-fptbl DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:247 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47d4b722-8199-4e8e-87bf-915e1ac409f5/volumes/kubernetes.io~projected/kube-api-access-lzswc DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8eeb180f70ec034f4e3eea020d257929f75b83377a99352b01db8234a02d3ca7/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a957db89-5022-4877-aa05-3713401881ed/volumes/kubernetes.io~projected/kube-api-access-l6k56 DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a43e9a82fbbc81b5c0e127f3f8ed690d7fa159485006e14ff5b543613089062d/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ec5f69d636b214266e4afa421e5792d890a81db8f8e8f5db5cad39d0e21e8c4/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/64491765ef85d4cabbae65ed27e83cbb6188588cf0895952d26d16d7f6071fed/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b5e3f9c-61e6-4468-9891-afdeb408caf7/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d/volumes/kubernetes.io~projected/kube-api-access-knqzf DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cda5dcc5-e5c6-4e0a-9acb-6191475f552c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9bcc2a31-3406-4199-b08d-1ad8817a5cc2/volumes/kubernetes.io~projected/kube-api-access-xwwgj DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f5c68fab-89c6-485c-a250-cb958115e5a2/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2cb0ab42-11a8-493c-8218-4755fb32e543/volumes/kubernetes.io~projected/kube-api-access-rhdgp DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb/volumes/kubernetes.io~projected/kube-api-access-657qr DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/709a3960720e1f59c1ae8a7a1f387366ff04f69fa827dec3ba5f6673185ab7c8/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa/volumes/kubernetes.io~projected/kube-api-access-7dbdb DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e49095abaa2842e614824b2264069e2370b832b0454c1cedf7577eb10be3f11b/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/622aafbf0dd8f040baf632757fa42b94b6b3f2b35c29f708587332a38c5e5f82/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/caa2b56f9caef11de2206811cb5d6524aba33a4d8e287fd1f6583f45e991c4fd/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3313df2c-6c0b-44d8-a356-8da87e1d620a/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/056de880-e0a6-4b20-805f-669a017b3d1d/volumes/kubernetes.io~projected/kube-api-access-shnrb DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cda5dcc5-e5c6-4e0a-9acb-6191475f552c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/f714c9b3-ccf2-4dab-9220-321c5b4258d0/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:237 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1f4e4ae7383bb270f7f543a25a3d55244964cf83e1ff541f37b644e7bed45f0d/userdata/shm DeviceMajor:0 DeviceMinor:251 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a3ad5fa672b2a2db023d36b971a7bf40bad4946facdfab4d6eabb31ff4bf5eba/userdata/shm DeviceMajor:0 DeviceMinor:273 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab67e784a14ff154144d603e90c8fce7bff0153a910b8f931121c80a8eba8ab9/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:248 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fdf1fa41-a03b-4a64-ba80-4c7a2825e259/volumes/kubernetes.io~projected/kube-api-access-lblgh DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d3fe1d96745eb4e733fc9f89eebf2eda108501e387f4349ffb9579ffdfa4c56/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45f57db50ba1789cd6bb63193217cd822742e83154a9cfeb674d13ce7e0b0056/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52b495cc-bf15-4438-8762-aab455e12701/volumes/kubernetes.io~projected/kube-api-access-lwdf9 DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:250 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bcc2a31-3406-4199-b08d-1ad8817a5cc2/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cda5dcc5-e5c6-4e0a-9acb-6191475f552c/volumes/kubernetes.io~projected/kube-api-access-zcl8m DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b5e3f9c-61e6-4468-9891-afdeb408caf7/volumes/kubernetes.io~projected/kube-api-access-ltgxk DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/47d4b722-8199-4e8e-87bf-915e1ac409f5/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b13e823829260663bb49df1ca87e7d3309632c077cbd5b794617aaadedfd598b/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/47d4b722-8199-4e8e-87bf-915e1ac409f5/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b516d30a0d2610ab2354787c4667cf4555c8a1ce3ef2c105dff044a87d227c6/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/876a361fa14fd14406a22cd3e2e4150563d7a08a7131204279c32ab1bb2fc1fc/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5dd6113daff0a3d568c8b28c559270b7619b13e6d7490bbad4a71a3e5c3d9183/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c48f7097-d8ba-4a86-9fe7-d56739a332f3/volumes/kubernetes.io~projected/kube-api-access-zjjdc DeviceMajor:0 DeviceMinor:249 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:269 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7cacd9fe2ab54e5a06edd5fd7347a283dc0f3a68502f21d14fa868da3a228bdc/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1f4e4ae7383bb27 MacAddress:3a:a9:66:bc:a0:b2 Speed:10000 Mtu:8901} {Name:5dd6113daff0a3d MacAddress:a2:26:16:8b:ac:46 Speed:10000 Mtu:8901} {Name:709a3960720e1f5 MacAddress:6a:4e:d8:62:23:52 Speed:10000 Mtu:8901} {Name:7cacd9fe2ab54e5 MacAddress:5e:09:8d:e4:fe:e8 Speed:10000 Mtu:8901} {Name:a3ad5fa672b2a2d MacAddress:e6:76:71:f8:fa:01 Speed:10000 Mtu:8901} {Name:a43e9a82fbbc81b MacAddress:36:5e:0a:fc:48:a0 Speed:10000 Mtu:8901} {Name:ab67e784a14ff15 MacAddress:6e:25:91:8f:cb:a0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:c4:39:99:dd Speed:0 Mtu:9001} {Name:br-int MacAddress:c2:6f:64:24:8c:1a Speed:0 Mtu:8901} {Name:caa2b56f9caef11 MacAddress:26:95:f1:c4:90:2d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:c4:39:99:dd Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:de:21:f2:08:76:f4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:72:c8:12:6c:7e:47 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.090446    4900 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.090489    4900 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.090773    4900 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.090814    4900 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-83.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.090936    4900 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.090944    4900 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.090955    4900 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.090969    4900 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.091056    4900 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.091211    4900 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.091323    4900 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.091356    4900 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.091369    4900 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.091379    4900 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.091388    4900 kubelet.go:405] "Adding apiserver pod source"
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.091396    4900 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.091929    4900 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.091938    4900 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:07:27.092881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.092381    4900 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 03:07:27.098957 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.098940    4900 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 03:07:27.098957 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.098955    4900 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:07:27.099094 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099055    4900 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 03:07:27.099245 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099234    4900 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 03:07:27.099280 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099263    4900 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 03:07:27.099361 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099352    4900 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 03:07:27.099401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099366    4900 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 03:07:27.099401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099373    4900 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 03:07:27.099401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099379    4900 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 03:07:27.099401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099384    4900 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 03:07:27.099401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099390    4900 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 03:07:27.099401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099395    4900 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 03:07:27.099401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099401    4900 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 03:07:27.099603 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099408    4900 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 03:07:27.099603 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099414    4900 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 03:07:27.099603 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099423    4900 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 03:07:27.099603 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099433    4900 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 03:07:27.099603 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099456    4900 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 03:07:27.099603 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.099462    4900 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 03:07:27.102869 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.102828    4900 server.go:1263] "Started kubelet"
Oct 31 03:07:27.102961 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.102900    4900 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 03:07:27.103350 ip-10-0-134-83 systemd[1]: Started Kubernetes Kubelet.
Oct 31 03:07:27.104466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.103157    4900 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 03:07:27.104596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.104578    4900 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 03:07:27.104750 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.104578    4900 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 03:07:27.105464 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.105318    4900 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 03:07:27.109037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.109020    4900 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 03:07:27.109134 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.109060    4900 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 03:07:27.109134 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.109115    4900 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:01:19 +0000 UTC" deadline="2028-04-08 09:21:07.963761839 +0000 UTC"
Oct 31 03:07:27.109235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.109135    4900 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12606h13m40.854629773s"
Oct 31 03:07:27.109235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.109158    4900 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 03:07:27.109235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.109174    4900 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 03:07:27.109235 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:07:27.109177    4900 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:07:27.109235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.109213    4900 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 03:07:27.111028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.110990    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa" volumeName="kubernetes.io/configmap/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-serviceca" seLinuxMountContext=""
Oct 31 03:07:27.111028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111027    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdc104e8-81dd-4bef-8dee-e28bf5c7a10d" volumeName="kubernetes.io/projected/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-api-access-knqzf" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111036    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" volumeName="kubernetes.io/projected/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-kube-api-access-lblgh" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111045    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056de880-e0a6-4b20-805f-669a017b3d1d" volumeName="kubernetes.io/empty-dir/056de880-e0a6-4b20-805f-669a017b3d1d-tmp-dir" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111054    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="569a2ae0-d9c0-4f18-a27c-14167c5ccdcb" volumeName="kubernetes.io/configmap/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-config-volume" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111061    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b5e3f9c-61e6-4468-9891-afdeb408caf7" volumeName="kubernetes.io/projected/8b5e3f9c-61e6-4468-9891-afdeb408caf7-kube-api-access-ltgxk" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111068    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90b722fb-cc4e-4c62-877b-db0df5c5d16c" volumeName="kubernetes.io/configmap/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-daemon-config" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111075    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90b722fb-cc4e-4c62-877b-db0df5c5d16c" volumeName="kubernetes.io/configmap/90b722fb-cc4e-4c62-877b-db0df5c5d16c-cni-binary-copy" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111084    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a957db89-5022-4877-aa05-3713401881ed" volumeName="kubernetes.io/configmap/a957db89-5022-4877-aa05-3713401881ed-iptables-alerter-script" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111092    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48f7097-d8ba-4a86-9fe7-d56739a332f3" volumeName="kubernetes.io/configmap/c48f7097-d8ba-4a86-9fe7-d56739a332f3-registry-certificates" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111101    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47d4b722-8199-4e8e-87bf-915e1ac409f5" volumeName="kubernetes.io/empty-dir/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-tuned" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111108    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47d4b722-8199-4e8e-87bf-915e1ac409f5" volumeName="kubernetes.io/projected/47d4b722-8199-4e8e-87bf-915e1ac409f5-kube-api-access-lzswc" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111115    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52b495cc-bf15-4438-8762-aab455e12701" volumeName="kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111122    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48f7097-d8ba-4a86-9fe7-d56739a332f3" volumeName="kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-kube-api-access-zjjdc" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111130    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="056de880-e0a6-4b20-805f-669a017b3d1d" volumeName="kubernetes.io/projected/056de880-e0a6-4b20-805f-669a017b3d1d-kube-api-access-shnrb" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111137    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="569a2ae0-d9c0-4f18-a27c-14167c5ccdcb" volumeName="kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111150    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bcc2a31-3406-4199-b08d-1ad8817a5cc2" volumeName="kubernetes.io/projected/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-kube-api-access-xwwgj" seLinuxMountContext=""
Oct 31 03:07:27.111155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111157    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cda5dcc5-e5c6-4e0a-9acb-6191475f552c" volumeName="kubernetes.io/projected/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-kube-api-access-zcl8m" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111172    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5c68fab-89c6-485c-a250-cb958115e5a2" volumeName="kubernetes.io/configmap/f5c68fab-89c6-485c-a250-cb958115e5a2-konnectivity-ca" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111179    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f714c9b3-ccf2-4dab-9220-321c5b4258d0" volumeName="kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111186    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" volumeName="kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111193    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3313df2c-6c0b-44d8-a356-8da87e1d620a" volumeName="kubernetes.io/projected/3313df2c-6c0b-44d8-a356-8da87e1d620a-kube-api-access-fptbl" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111200    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3313df2c-6c0b-44d8-a356-8da87e1d620a" volumeName="kubernetes.io/secret/3313df2c-6c0b-44d8-a356-8da87e1d620a-signing-key" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111207    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a957db89-5022-4877-aa05-3713401881ed" volumeName="kubernetes.io/projected/a957db89-5022-4877-aa05-3713401881ed-kube-api-access-l6k56" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111215    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48f7097-d8ba-4a86-9fe7-d56739a332f3" volumeName="kubernetes.io/configmap/c48f7097-d8ba-4a86-9fe7-d56739a332f3-trusted-ca" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111222    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa" volumeName="kubernetes.io/projected/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-kube-api-access-7dbdb" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111229    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" volumeName="kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cni-binary-copy" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111236    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" volumeName="kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111247    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cb0ab42-11a8-493c-8218-4755fb32e543" volumeName="kubernetes.io/projected/2cb0ab42-11a8-493c-8218-4755fb32e543-kube-api-access-rhdgp" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111254    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48f7097-d8ba-4a86-9fe7-d56739a332f3" volumeName="kubernetes.io/empty-dir/c48f7097-d8ba-4a86-9fe7-d56739a332f3-ca-trust-extracted" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111261    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cda5dcc5-e5c6-4e0a-9acb-6191475f552c" volumeName="kubernetes.io/secret/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111268    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5c68fab-89c6-485c-a250-cb958115e5a2" volumeName="kubernetes.io/secret/f5c68fab-89c6-485c-a250-cb958115e5a2-agent-certs" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111275    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdc104e8-81dd-4bef-8dee-e28bf5c7a10d" volumeName="kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111282    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="569a2ae0-d9c0-4f18-a27c-14167c5ccdcb" volumeName="kubernetes.io/empty-dir/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-tmp-dir" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111290    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bcc2a31-3406-4199-b08d-1ad8817a5cc2" volumeName="kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111296    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cda5dcc5-e5c6-4e0a-9acb-6191475f552c" volumeName="kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-env-overrides" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111303    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdc104e8-81dd-4bef-8dee-e28bf5c7a10d" volumeName="kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111310    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdc104e8-81dd-4bef-8dee-e28bf5c7a10d" volumeName="kubernetes.io/empty-dir/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-data-volume" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111317    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3313df2c-6c0b-44d8-a356-8da87e1d620a" volumeName="kubernetes.io/configmap/3313df2c-6c0b-44d8-a356-8da87e1d620a-signing-cabundle" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111324    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="47d4b722-8199-4e8e-87bf-915e1ac409f5" volumeName="kubernetes.io/empty-dir/47d4b722-8199-4e8e-87bf-915e1ac409f5-tmp" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111332    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="569a2ae0-d9c0-4f18-a27c-14167c5ccdcb" volumeName="kubernetes.io/projected/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-kube-api-access-657qr" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111340    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b5e3f9c-61e6-4468-9891-afdeb408caf7" volumeName="kubernetes.io/secret/8b5e3f9c-61e6-4468-9891-afdeb408caf7-cert" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111346    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90b722fb-cc4e-4c62-877b-db0df5c5d16c" volumeName="kubernetes.io/projected/90b722fb-cc4e-4c62-877b-db0df5c5d16c-kube-api-access-xlbbg" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111354    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48f7097-d8ba-4a86-9fe7-d56739a332f3" volumeName="kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-registry-tls" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111360    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48f7097-d8ba-4a86-9fe7-d56739a332f3" volumeName="kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-bound-sa-token" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111367    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48f7097-d8ba-4a86-9fe7-d56739a332f3" volumeName="kubernetes.io/secret/c48f7097-d8ba-4a86-9fe7-d56739a332f3-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111374    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c48f7097-d8ba-4a86-9fe7-d56739a332f3" volumeName="kubernetes.io/secret/c48f7097-d8ba-4a86-9fe7-d56739a332f3-installation-pull-secrets" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111381    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cda5dcc5-e5c6-4e0a-9acb-6191475f552c" volumeName="kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovnkube-config" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111389    4900 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cda5dcc5-e5c6-4e0a-9acb-6191475f552c" volumeName="kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111396    4900 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 03:07:27.111644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.111401    4900 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 03:07:27.113004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.112443    4900 factory.go:153] Registering CRI-O factory
Oct 31 03:07:27.113004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.112461    4900 factory.go:223] Registration of the crio container factory successfully
Oct 31 03:07:27.113004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.112513    4900 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 03:07:27.113004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.112522    4900 factory.go:55] Registering systemd factory
Oct 31 03:07:27.113004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.112529    4900 factory.go:223] Registration of the systemd container factory successfully
Oct 31 03:07:27.113004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.112550    4900 factory.go:103] Registering Raw factory
Oct 31 03:07:27.113004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.112564    4900 manager.go:1199] Started watching for new ooms in manager
Oct 31 03:07:27.113004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.112914    4900 manager.go:319] Starting recovery of all containers
Oct 31 03:07:27.125060 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.125039    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:07:27.125215 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.125195    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:07:27.127840 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.127813    4900 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 03:07:27.144675 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.144643    4900 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 03:07:27.144675 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.144671    4900 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 03:07:27.144993 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.144698    4900 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 03:07:27.144993 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:07:27.144735    4900 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 03:07:27.155052 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.155024    4900 generic.go:356] "Generic (PLEG): container finished" podID="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" containerID="0cd137a0436c0714916c96d907fe0e464a786c195f0d53dfcd7965fda9a6d07f" exitCode=0
Oct 31 03:07:27.155052 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.155049    4900 generic.go:356] "Generic (PLEG): container finished" podID="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" containerID="e70e374d67099e25ab8a1b859e3915ae7119c2476703454dacc84a61b526eb81" exitCode=0
Oct 31 03:07:27.155213 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.155058    4900 generic.go:356] "Generic (PLEG): container finished" podID="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" containerID="e15b951ac0a0c8746d765137d12b4a19536c4d5c5833ddfaf19d74e8e4fb0fd8" exitCode=0
Oct 31 03:07:27.155213 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.155065    4900 generic.go:356] "Generic (PLEG): container finished" podID="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" containerID="080ecbcedf4234973f699feb56de24640fe2a48f208da8c94b64e272e52752f6" exitCode=0
Oct 31 03:07:27.155213 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.155072    4900 generic.go:356] "Generic (PLEG): container finished" podID="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" containerID="308c480d5b47124a0fa0c9fa4415089f36cbaf5b44f19c85f442ca49a7d4fbd7" exitCode=0
Oct 31 03:07:27.155213 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.155079    4900 generic.go:356] "Generic (PLEG): container finished" podID="fdf1fa41-a03b-4a64-ba80-4c7a2825e259" containerID="9655ec86740d5d15396b58e641380c155a1dadc3baff0e06162bb478b0814200" exitCode=0
Oct 31 03:07:27.159099 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.159074    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:07:27.159424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.159395    4900 generic.go:356] "Generic (PLEG): container finished" podID="cda5dcc5-e5c6-4e0a-9acb-6191475f552c" containerID="e89a6744ed4fe5f9ef26d0cc49124150bd9a6f0792d62f25452d975faa0505ed" exitCode=1
Oct 31 03:07:27.181414 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.181381    4900 generic.go:356] "Generic (PLEG): container finished" podID="9d194eddb65e9ecc3707084f3a0166db" containerID="b239b72aa94ef50bd34a6618c5559ec7477d41b2365777bd8c89e5dbf4403910" exitCode=0
Oct 31 03:07:27.186432 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.186415    4900 manager.go:324] Recovery completed
Oct 31 03:07:27.200665 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.200643    4900 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:07:27.202735 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.202718    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:07:27.202834 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.202746    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:07:27.202834 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.202757    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:07:27.204615 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.204598    4900 cpu_manager.go:226] "Starting" policy="none"
Oct 31 03:07:27.204615 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.204613    4900 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 03:07:27.204743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.204619    4900 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:07:27.204743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.204629    4900 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 03:07:27.204820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.204759    4900 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 03:07:27.204820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.204770    4900 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 03:07:27.204820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.204799    4900 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 03:07:27.204820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.204808    4900 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 03:07:27.204820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.204819    4900 policy_none.go:50] "Start"
Oct 31 03:07:27.204975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.204828    4900 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 03:07:27.204975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.204838    4900 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 03:07:27.205033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.205019    4900 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 03:07:27.205033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.205027    4900 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 03:07:27.205033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.205033    4900 policy_none.go:44] "Start"
Oct 31 03:07:27.205405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.205391    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:07:27.205455 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.205419    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:07:27.205455 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.205434    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:07:27.209280 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:07:27.209257    4900 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:07:27.211408 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.211380    4900 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 03:07:27.213956 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.213944    4900 manager.go:341] "Starting Device Plugin manager"
Oct 31 03:07:27.214007 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:07:27.213968    4900 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 03:07:27.214007 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.213978    4900 server.go:87] "Starting device plugin registration server"
Oct 31 03:07:27.214253 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.214240    4900 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 03:07:27.214326 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.214253    4900 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 03:07:27.214376 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.214338    4900 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 03:07:27.214469 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.214431    4900 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 03:07:27.214564 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.214478    4900 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 03:07:27.214564 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.214485    4900 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 03:07:27.214666 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.214536    4900 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:07:27.214463786Z","UUID":"662a700e-c625-4ff4-b32d-bd210a742d38","Handler":null,"Name":"","Endpoint":""}
Oct 31 03:07:27.215901 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.215873    4900 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 03:07:27.215901 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.215896    4900 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 03:07:27.216042 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:07:27.216022    4900 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 03:07:27.216075 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:07:27.216057    4900 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-83.us-west-2.compute.internal\" not found"
Oct 31 03:07:27.245711 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.245669    4900 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"]
Oct 31 03:07:27.245874 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.245817    4900 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:07:27.246626 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.246609    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:07:27.246726 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.246640    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:07:27.246726 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.246655    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:07:27.246980 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.246965    4900 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:07:27.247099 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.247084    4900 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:07:27.247752 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.247731    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:07:27.247847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.247764    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:07:27.247847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.247766    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:07:27.247847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.247780    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:07:27.247847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.247798    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:07:27.247847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.247814    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:07:27.247994 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:07:27.247967    4900 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found" node="ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.248149 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.248113    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal" event={"ID":"584715508ebcb798df8a2f82469ae3b6","Type":"ContainerStarted","Data":"e8991270f7e891d11886dc413f42b5d90d0e165389ed2b8530e1d93a7b23351a"}
Oct 31 03:07:27.248182 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.248162    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal" event={"ID":"584715508ebcb798df8a2f82469ae3b6","Type":"ContainerStarted","Data":"8eeb180f70ec034f4e3eea020d257929f75b83377a99352b01db8234a02d3ca7"}
Oct 31 03:07:27.248210 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.248191    4900 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:07:27.248210 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.248204    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal" event={"ID":"9d194eddb65e9ecc3707084f3a0166db","Type":"ContainerStarted","Data":"68a8a97288afcc8a244282947d6db9e9e566ae7cdc2c1c105662067de40cf0b4"}
Oct 31 03:07:27.248294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.248224    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal" event={"ID":"9d194eddb65e9ecc3707084f3a0166db","Type":"ContainerDied","Data":"b239b72aa94ef50bd34a6618c5559ec7477d41b2365777bd8c89e5dbf4403910"}
Oct 31 03:07:27.248294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.248244    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal" event={"ID":"9d194eddb65e9ecc3707084f3a0166db","Type":"ContainerStarted","Data":"45f57db50ba1789cd6bb63193217cd822742e83154a9cfeb674d13ce7e0b0056"}
Oct 31 03:07:27.248966 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.248946    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:07:27.249055 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.248976    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:07:27.249055 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.248989    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:07:27.249158 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:07:27.249143    4900 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-83.us-west-2.compute.internal\" not found" node="ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.314713 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.314683    4900 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:07:27.315677 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.315656    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:07:27.315760 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.315693    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:07:27.315760 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.315704    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:07:27.315760 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.315722    4900 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.334350 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.334332    4900 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 03:07:27.339141 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.339119    4900 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:07:27.339231 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.339147    4900 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:07:27.339231 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.339154    4900 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:07:27.381468 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.381435    4900 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:07:27.409756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.409732    4900 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.430024 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.429997    4900 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.430121 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.430073    4900 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.430660 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.430632    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeNotReady"
Oct 31 03:07:27.430760 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.430668    4900 setters.go:546] "Node became not ready" node="ip-10-0-134-83.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T03:07:27Z","lastTransitionTime":"2026-10-31T03:07:27Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 31 03:07:27.434469 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.434445    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d194eddb65e9ecc3707084f3a0166db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"9d194eddb65e9ecc3707084f3a0166db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.434549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.434474    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d194eddb65e9ecc3707084f3a0166db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"9d194eddb65e9ecc3707084f3a0166db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.434549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.434492    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/584715508ebcb798df8a2f82469ae3b6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"584715508ebcb798df8a2f82469ae3b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.468114 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.468084    4900 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:07:27.468249 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:07:27.468136    4900 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.468249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.468150    4900 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.484313 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.484290    4900 nodeinfomanager.go:471] existing CSINode "ip-10-0-134-83.us-west-2.compute.internal" is owned by different node (oldNodeID="2e029cca-c98a-4159-9b1b-023dea3ad549", newNodeID=""), cleaning up...
Oct 31 03:07:27.534809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.534706    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d194eddb65e9ecc3707084f3a0166db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"9d194eddb65e9ecc3707084f3a0166db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.534809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.534748    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d194eddb65e9ecc3707084f3a0166db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"9d194eddb65e9ecc3707084f3a0166db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.535003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.534828    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d194eddb65e9ecc3707084f3a0166db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"9d194eddb65e9ecc3707084f3a0166db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.535003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.534873    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d194eddb65e9ecc3707084f3a0166db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"9d194eddb65e9ecc3707084f3a0166db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.535003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.534908    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/584715508ebcb798df8a2f82469ae3b6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"584715508ebcb798df8a2f82469ae3b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.535003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.534955    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/584715508ebcb798df8a2f82469ae3b6-config\") pod \"kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal\" (UID: \"584715508ebcb798df8a2f82469ae3b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.570036 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.570009    4900 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:07:27.570179 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:07:27.570056    4900 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:27.694987 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.694959    4900 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-83.us-west-2.compute.internal" event="NodeReady"
Oct 31 03:07:27.695159 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:27.695050    4900 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 03:07:28.091757 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.091717    4900 apiserver.go:52] "Watching apiserver"
Oct 31 03:07:28.188169 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.188135    4900 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 03:07:28.188633 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.188607    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-ndfgt","openshift-ovn-kubernetes/ovnkube-node-xm5zs","openshift-dns/dns-default-whz62","openshift-insights/insights-runtime-extractor-8z98n","openshift-multus/network-metrics-daemon-58h48","openshift-network-operator/iptables-alerter-28rvl","openshift-cluster-node-tuning-operator/tuned-fwmjk","openshift-image-registry/image-registry-ccdffd8bb-vmv6b","openshift-ingress-canary/ingress-canary-p5zp9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-mmv7s","kube-system/konnectivity-agent-c7b69","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh","openshift-image-registry/node-ca-sdwgq","openshift-service-ca/service-ca-5fffc97bdf-r89cn","openshift-multus/multus-6vx62","openshift-multus/multus-additional-cni-plugins-brs5l","openshift-dns/node-resolver-4fpkl","kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"]
Oct 31 03:07:28.190242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.190222    4900 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:28.190343 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.190280    4900 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:28.210624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.210597    4900 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 03:07:28.239806 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.239765    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 03:07:28.239937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.239808    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 03:07:28.239937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.239859    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 03:07:28.239937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.239926    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 03:07:28.240089 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.239986    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 03:07:28.240089 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240025    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 03:07:28.240089 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240078    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 03:07:28.240216 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240199    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 03:07:28.240264 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240213    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 03:07:28.240264 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240228    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:07:28.240264 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240252    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 03:07:28.240264 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240260    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 03:07:28.240454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240282    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:07:28.240454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240314    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 03:07:28.240454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240319    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c48f7097-d8ba-4a86-9fe7-d56739a332f3-registry-certificates\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.240454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240354    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 03:07:28.240454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240350    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjjdc\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-kube-api-access-zjjdc\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.240454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240379    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 03:07:28.240454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240405    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.240454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240438    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzswc\" (UniqueName: \"kubernetes.io/projected/47d4b722-8199-4e8e-87bf-915e1ac409f5-kube-api-access-lzswc\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.240765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240458    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 03:07:28.240765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240470    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5c68fab-89c6-485c-a250-cb958115e5a2-agent-certs\") pod \"konnectivity-agent-c7b69\" (UID: \"f5c68fab-89c6-485c-a250-cb958115e5a2\") " pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:07:28.240765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240504    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 03:07:28.240765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240523    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.240765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240573    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovn-node-metrics-cert\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.240765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240616    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:07:28.240765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240634    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 03:07:28.240765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240654    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-system-cni-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.240765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240690    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c48f7097-d8ba-4a86-9fe7-d56739a332f3-registry-certificates\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.240765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240756    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shnrb\" (UniqueName: \"kubernetes.io/projected/056de880-e0a6-4b20-805f-669a017b3d1d-kube-api-access-shnrb\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:07:28.241156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240800    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a957db89-5022-4877-aa05-3713401881ed-host-slash\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:07:28.241156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240821    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47d4b722-8199-4e8e-87bf-915e1ac409f5-tmp\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.241156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240916    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f714c9b3-ccf2-4dab-9220-321c5b4258d0-kubelet-config\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:07:28.241156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240953    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-system-cni-dir\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.241156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.240984    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/47d4b722-8199-4e8e-87bf-915e1ac409f5-tmp\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.241156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.241011    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.241156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.241037    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-cni-multus\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.241452 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.241342    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056de880-e0a6-4b20-805f-669a017b3d1d-tmp-dir\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:07:28.241452 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.241403    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-data-volume\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:28.241452 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.241434    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-cnibin\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.241574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.241465    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-var-lib-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.241574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.241478    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056de880-e0a6-4b20-805f-669a017b3d1d-tmp-dir\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:07:28.241574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.241499    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-etc-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.241574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.241531    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-cni-netd\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.241574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.241560    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657qr\" (UniqueName: \"kubernetes.io/projected/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-kube-api-access-657qr\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:07:28.241856 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.241836    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 03:07:28.241991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.241972    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-data-volume\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:28.242048 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.241983    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-bdmwz\""
Oct 31 03:07:28.242048 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242021    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8l9rs\""
Oct 31 03:07:28.242145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242005    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2wckk\""
Oct 31 03:07:28.242201 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242182    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwwgj\" (UniqueName: \"kubernetes.io/projected/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-kube-api-access-xwwgj\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:07:28.242254 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242222    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 03:07:28.242254 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242231    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysconfig\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.242343 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242262    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-kubernetes\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.242391 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242367    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-var-lib-kubelet\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.242391 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242372    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lsfgs\""
Oct 31 03:07:28.242485 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242398    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5c68fab-89c6-485c-a250-cb958115e5a2-konnectivity-ca\") pod \"konnectivity-agent-c7b69\" (UID: \"f5c68fab-89c6-485c-a250-cb958115e5a2\") " pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:07:28.242485 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242429    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/90b722fb-cc4e-4c62-877b-db0df5c5d16c-cni-binary-copy\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.242485 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242459    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-netns\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.242616 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242486    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-slash\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.242616 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242608    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gtdck\""
Oct 31 03:07:28.242847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242808    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f5c68fab-89c6-485c-a250-cb958115e5a2-konnectivity-ca\") pod \"konnectivity-agent-c7b69\" (UID: \"f5c68fab-89c6-485c-a250-cb958115e5a2\") " pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:07:28.243003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242973    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/90b722fb-cc4e-4c62-877b-db0df5c5d16c-cni-binary-copy\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.243003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.242608    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-k8s-cni-cncf-io\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.243111 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243039    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-conf-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.243111 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243078    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbbg\" (UniqueName: \"kubernetes.io/projected/90b722fb-cc4e-4c62-877b-db0df5c5d16c-kube-api-access-xlbbg\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.243202 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243110    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dbdb\" (UniqueName: \"kubernetes.io/projected/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-kube-api-access-7dbdb\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:07:28.243202 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243144    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-kubelet\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.243202 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243176    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.243320 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243202    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcl8m\" (UniqueName: \"kubernetes.io/projected/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-kube-api-access-zcl8m\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.243320 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243232    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fptbl\" (UniqueName: \"kubernetes.io/projected/3313df2c-6c0b-44d8-a356-8da87e1d620a-kube-api-access-fptbl\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:07:28.243320 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243263    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovnkube-config\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.243440 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243295    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovnkube-script-lib\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.243440 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243367    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c48f7097-d8ba-4a86-9fe7-d56739a332f3-ca-trust-extracted\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.243440 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243404    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c48f7097-d8ba-4a86-9fe7-d56739a332f3-trusted-ca\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.243569 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243434    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c48f7097-d8ba-4a86-9fe7-d56739a332f3-installation-pull-secrets\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.243569 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243514    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-bound-sa-token\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.243569 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243549    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-sys-fs\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.243697 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243592    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b5e3f9c-61e6-4468-9891-afdeb408caf7-cert\") pod \"ingress-canary-p5zp9\" (UID: \"8b5e3f9c-61e6-4468-9891-afdeb408caf7\") " pod="openshift-ingress-canary/ingress-canary-p5zp9"
Oct 31 03:07:28.243697 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243619    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-daemon-config\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.243697 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243657    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-etc-kubernetes\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.243844 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243692    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwdf9\" (UniqueName: \"kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9\") pod \"network-check-target-mmv7s\" (UID: \"52b495cc-bf15-4438-8762-aab455e12701\") " pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:07:28.243844 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243725    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-run-ovn-kubernetes\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.243987 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.243966    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c48f7097-d8ba-4a86-9fe7-d56739a332f3-ca-trust-extracted\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.244079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244060    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysctl-conf\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.244144 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244104    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-tuned\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.244144 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244135    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-multus-certs\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.244231 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244159    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-systemd\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.244231 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244196    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-host\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.244313 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244274    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9g6gp\""
Oct 31 03:07:28.244361 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244328    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-tuned\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.244495 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244476    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-daemon-config\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.244544 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244523    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-kubelet\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.244596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244562    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-run-netns\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.244644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244606    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c48f7097-d8ba-4a86-9fe7-d56739a332f3-image-registry-private-configuration\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.244701 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244640    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhdgp\" (UniqueName: \"kubernetes.io/projected/2cb0ab42-11a8-493c-8218-4755fb32e543-kube-api-access-rhdgp\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.244753 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244745    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-modprobe-d\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.244831 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244780    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysctl-d\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.244879 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244849    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:28.244923 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244878    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cni-binary-copy\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.244976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244923    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-socket-dir-parent\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.245026 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.244976    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-hostroot\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.245026 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.245013    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-host\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:07:28.245110 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.245085    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:28.245270 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.245237    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.245371 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.245352    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cni-binary-copy\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.245485 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.245465    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-registration-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.245694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.245639    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a957db89-5022-4877-aa05-3713401881ed-iptables-alerter-script\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:07:28.245817 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.245719    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-run\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.245817 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.245751    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-lib-modules\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.245945 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.245854    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:07:28.245945 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.245899    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3313df2c-6c0b-44d8-a356-8da87e1d620a-signing-key\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:07:28.245945 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.245936    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/056de880-e0a6-4b20-805f-669a017b3d1d-hosts-file\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:07:28.246094 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.245996    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.246156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246093    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-device-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.246156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246095    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a957db89-5022-4877-aa05-3713401881ed-iptables-alerter-script\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:07:28.246156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246140    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6k56\" (UniqueName: \"kubernetes.io/projected/a957db89-5022-4877-aa05-3713401881ed-kube-api-access-l6k56\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:07:28.246273 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246207    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:07:28.246611 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246594    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltgxk\" (UniqueName: \"kubernetes.io/projected/8b5e3f9c-61e6-4468-9891-afdeb408caf7-kube-api-access-ltgxk\") pod \"ingress-canary-p5zp9\" (UID: \"8b5e3f9c-61e6-4468-9891-afdeb408caf7\") " pod="openshift-ingress-canary/ingress-canary-p5zp9"
Oct 31 03:07:28.246664 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246625    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knqzf\" (UniqueName: \"kubernetes.io/projected/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-api-access-knqzf\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:28.246664 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246643    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-cni-bin\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.246664 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246659    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-socket-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.246839 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246775    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-crio-socket\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:28.246839 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246821    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-cni-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.246959 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246846    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-serviceca\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:07:28.246959 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246861    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-systemd-units\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.246959 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246880    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-systemd\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.246959 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246926    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-node-log\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.246959 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.246956    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-config-volume\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:07:28.247221 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247073    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-sys\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.247221 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247109    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cnibin\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.247221 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247152    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lblgh\" (UniqueName: \"kubernetes.io/projected/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-kube-api-access-lblgh\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.247221 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247169    4900 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:07:28.247221 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:07:28.247219    4900 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:28.247494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247233    4900 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:07:28.247494 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:07:28.247284    4900 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-83.us-west-2.compute.internal"
Oct 31 03:07:28.247494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247297    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3313df2c-6c0b-44d8-a356-8da87e1d620a-signing-cabundle\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:07:28.247494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247333    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-ovn\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.247494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247358    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-cni-bin\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.247494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247395    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.247743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247494    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-env-overrides\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.247743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247505    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-serviceca\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:07:28.247743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247526    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f714c9b3-ccf2-4dab-9220-321c5b4258d0-dbus\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:07:28.247743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247543    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-log-socket\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.247743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247598    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-tmp-dir\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:07:28.247743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247635    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-registry-tls\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.247743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.247654    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-tmp-dir\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:07:28.251600 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.251586    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f5c68fab-89c6-485c-a250-cb958115e5a2-agent-certs\") pod \"konnectivity-agent-c7b69\" (UID: \"f5c68fab-89c6-485c-a250-cb958115e5a2\") " pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:07:28.293010 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.292975    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzswc\" (UniqueName: \"kubernetes.io/projected/47d4b722-8199-4e8e-87bf-915e1ac409f5-kube-api-access-lzswc\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.293257 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.293237    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjjdc\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-kube-api-access-zjjdc\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.294743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.294719    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shnrb\" (UniqueName: \"kubernetes.io/projected/056de880-e0a6-4b20-805f-669a017b3d1d-kube-api-access-shnrb\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:07:28.294877 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.294860    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwwgj\" (UniqueName: \"kubernetes.io/projected/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-kube-api-access-xwwgj\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:07:28.294978 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.294963    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-657qr\" (UniqueName: \"kubernetes.io/projected/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-kube-api-access-657qr\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:07:28.298195 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.298166    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dbdb\" (UniqueName: \"kubernetes.io/projected/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-kube-api-access-7dbdb\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:07:28.298331 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.298314    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-bound-sa-token\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.298495 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.298477    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhdgp\" (UniqueName: \"kubernetes.io/projected/2cb0ab42-11a8-493c-8218-4755fb32e543-kube-api-access-rhdgp\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.298623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.298599    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbbg\" (UniqueName: \"kubernetes.io/projected/90b722fb-cc4e-4c62-877b-db0df5c5d16c-kube-api-access-xlbbg\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.298695 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.298674    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6k56\" (UniqueName: \"kubernetes.io/projected/a957db89-5022-4877-aa05-3713401881ed-kube-api-access-l6k56\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:07:28.298901 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.298882    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwdf9\" (UniqueName: \"kubernetes.io/projected/52b495cc-bf15-4438-8762-aab455e12701-kube-api-access-lwdf9\") pod \"network-check-target-mmv7s\" (UID: \"52b495cc-bf15-4438-8762-aab455e12701\") " pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:07:28.299937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.299920    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lblgh\" (UniqueName: \"kubernetes.io/projected/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-kube-api-access-lblgh\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.341115 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.341083    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 03:07:28.341250 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.341133    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 03:07:28.341250 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.341241    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 03:07:28.341392 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.341378    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 03:07:28.341508 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.341493    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 03:07:28.341585 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.341537    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 03:07:28.341638 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.341623    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 03:07:28.341869 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.341805    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 03:07:28.341869 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.341857    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 03:07:28.342043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.341988    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 03:07:28.342043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.342006    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 03:07:28.342163 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.342050    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 03:07:28.342163 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.342056    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 03:07:28.342163 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.342063    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 03:07:28.342163 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.342102    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 03:07:28.342640 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.342608    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 03:07:28.342809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.342759    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m6gdv\""
Oct 31 03:07:28.342809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.342769    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wjqqx\""
Oct 31 03:07:28.342809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.342804    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 03:07:28.342972 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.342835    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jbq7d\""
Oct 31 03:07:28.342972 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.342844    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xtr75\""
Oct 31 03:07:28.343058 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.342993    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 03:07:28.344055 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.344040    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovnkube-config\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.344262 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.344245    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-47gs9\""
Oct 31 03:07:28.344347 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.344299    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ffsgr\""
Oct 31 03:07:28.344347 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.344306    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 03:07:28.344347 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.344308    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-89z7n\""
Oct 31 03:07:28.344511 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.344493    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b5e3f9c-61e6-4468-9891-afdeb408caf7-cert\") pod \"ingress-canary-p5zp9\" (UID: \"8b5e3f9c-61e6-4468-9891-afdeb408caf7\") " pod="openshift-ingress-canary/ingress-canary-p5zp9"
Oct 31 03:07:28.344594 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.344509    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l8z7f\""
Oct 31 03:07:28.344594 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.344519    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 03:07:28.344594 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.344499    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sc79n\""
Oct 31 03:07:28.344594 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.344523    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 03:07:28.344594 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.344537    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-95c6m\""
Oct 31 03:07:28.344594 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.344558    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 03:07:28.344594 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.344578    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 03:07:28.344818 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.344600    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-z58pm\""
Oct 31 03:07:28.345236 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.345216    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcl8m\" (UniqueName: \"kubernetes.io/projected/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-kube-api-access-zcl8m\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.345380 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.345360    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c48f7097-d8ba-4a86-9fe7-d56739a332f3-image-registry-private-configuration\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.345572 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.345557    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:28.345715 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.345701    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.345770 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.345737    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:28.347070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.347054    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bcc2a31-3406-4199-b08d-1ad8817a5cc2-metrics-certs\") pod \"network-metrics-daemon-58h48\" (UID: \"9bcc2a31-3406-4199-b08d-1ad8817a5cc2\") " pod="openshift-multus/network-metrics-daemon-58h48"
Oct 31 03:07:28.347126 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.347056    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/3313df2c-6c0b-44d8-a356-8da87e1d620a-signing-key\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:07:28.347173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.347166    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-config-volume\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:07:28.348121 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348107    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-env-overrides\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.348238 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348223    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/3313df2c-6c0b-44d8-a356-8da87e1d620a-signing-cabundle\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:07:28.348293 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348277    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysctl-conf\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.348331 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348313    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-multus-certs\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.348367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348335    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-systemd\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.348416 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348377    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-systemd\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.348416 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348383    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-multus-certs\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.348416 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348397    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-host\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.348562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348418    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-kubelet\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.348562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348432    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-run-netns\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.348562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348445    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysctl-conf\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.348562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348448    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-modprobe-d\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.348562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348482    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fptbl\" (UniqueName: \"kubernetes.io/projected/3313df2c-6c0b-44d8-a356-8da87e1d620a-kube-api-access-fptbl\") pod \"service-ca-5fffc97bdf-r89cn\" (UID: \"3313df2c-6c0b-44d8-a356-8da87e1d620a\") " pod="openshift-service-ca/service-ca-5fffc97bdf-r89cn"
Oct 31 03:07:28.348562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348494    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-host\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.348562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348484    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-run-netns\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.348562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348486    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-kubelet\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.348562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348520    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysctl-d\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.348562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348549    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-socket-dir-parent\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348574    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-hostroot\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348596    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-socket-dir-parent\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348602    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-host\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348600    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-modprobe-d\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348628    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysctl-d\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348648    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa-host\") pod \"node-ca-sdwgq\" (UID: \"fd3a8dde-2755-44ac-9fc3-6bb5ddbaa9aa\") " pod="openshift-image-registry/node-ca-sdwgq"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348639    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-registration-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348685    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-registration-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348690    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-run\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348653    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-hostroot\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348710    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-lib-modules\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348722    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-run\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348734    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/056de880-e0a6-4b20-805f-669a017b3d1d-hosts-file\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348780    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348783    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-lib-modules\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348822    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-device-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348831    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348826    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/056de880-e0a6-4b20-805f-669a017b3d1d-hosts-file\") pod \"node-resolver-4fpkl\" (UID: \"056de880-e0a6-4b20-805f-669a017b3d1d\") " pod="openshift-dns/node-resolver-4fpkl"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348842    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-cni-bin\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348858    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-socket-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348865    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-device-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348878    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-crio-socket\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348890    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-cni-bin\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348897    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-cni-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348902    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-socket-dir\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348924    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-systemd-units\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348934    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-cni-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348949    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-systemd-units\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348967    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-crio-socket\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.348968    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-systemd\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349002    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-systemd\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349004    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-node-log\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349029    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-node-log\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349032    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-sys\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349076    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cnibin\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.349079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349093    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-sys\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349122    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltgxk\" (UniqueName: \"kubernetes.io/projected/8b5e3f9c-61e6-4468-9891-afdeb408caf7-kube-api-access-ltgxk\") pod \"ingress-canary-p5zp9\" (UID: \"8b5e3f9c-61e6-4468-9891-afdeb408caf7\") " pod="openshift-ingress-canary/ingress-canary-p5zp9"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349134    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-ovn\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349105    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-ovn\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349161    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-cnibin\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349185    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-cni-bin\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349210    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-cni-bin\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349231    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349267    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f714c9b3-ccf2-4dab-9220-321c5b4258d0-dbus\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349292    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-log-socket\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349306    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349319    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349334    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f714c9b3-ccf2-4dab-9220-321c5b4258d0-dbus\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349345    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-log-socket\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349349    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349376    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-system-cni-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349380    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-etc-selinux\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349401    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a957db89-5022-4877-aa05-3713401881ed-host-slash\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349429    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f714c9b3-ccf2-4dab-9220-321c5b4258d0-kubelet-config\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349413    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-tuning-conf-dir\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349448    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knqzf\" (UniqueName: \"kubernetes.io/projected/fdc104e8-81dd-4bef-8dee-e28bf5c7a10d-kube-api-access-knqzf\") pod \"insights-runtime-extractor-8z98n\" (UID: \"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d\") " pod="openshift-insights/insights-runtime-extractor-8z98n"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349465    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-system-cni-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349477    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f714c9b3-ccf2-4dab-9220-321c5b4258d0-kubelet-config\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349478    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a957db89-5022-4877-aa05-3713401881ed-host-slash\") pod \"iptables-alerter-28rvl\" (UID: \"a957db89-5022-4877-aa05-3713401881ed\") " pod="openshift-network-operator/iptables-alerter-28rvl"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349497    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-system-cni-dir\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349517    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-cni-multus\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349533    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-cnibin\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349554    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-var-lib-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349573    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-var-lib-cni-multus\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349579    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-etc-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349587    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-cnibin\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349556    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-system-cni-dir\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349592    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-var-lib-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349602    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-cni-netd\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349632    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-etc-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349669    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-cni-netd\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349671    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysconfig\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349688    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-sysconfig\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349703    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-kubernetes\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349737    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-var-lib-kubelet\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349744    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-etc-kubernetes\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349756    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-netns\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349771    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-slash\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349808    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/47d4b722-8199-4e8e-87bf-915e1ac409f5-var-lib-kubelet\") pod \"tuned-fwmjk\" (UID: \"47d4b722-8199-4e8e-87bf-915e1ac409f5\") " pod="openshift-cluster-node-tuning-operator/tuned-fwmjk"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349820    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-k8s-cni-cncf-io\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349838    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-netns\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349844    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-conf-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349873    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-host-run-k8s-cni-cncf-io\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349880    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-kubelet\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349854    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-slash\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349905    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349925    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-kubelet\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349908    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-multus-conf-dir\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349935    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-run-openvswitch\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349937    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-sys-fs\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349973    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-etc-kubernetes\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.349979    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2cb0ab42-11a8-493c-8218-4755fb32e543-sys-fs\") pod \"aws-ebs-csi-driver-node-ldhbh\" (UID: \"2cb0ab42-11a8-493c-8218-4755fb32e543\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ldhbh"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.350001    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-run-ovn-kubernetes\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.350015    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90b722fb-cc4e-4c62-877b-db0df5c5d16c-etc-kubernetes\") pod \"multus-6vx62\" (UID: \"90b722fb-cc4e-4c62-877b-db0df5c5d16c\") " pod="openshift-multus/multus-6vx62"
Oct 31 03:07:28.350362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.350052    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-host-run-ovn-kubernetes\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.352040 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.350960    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovn-node-metrics-cert\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.352040 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.350979    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/569a2ae0-d9c0-4f18-a27c-14167c5ccdcb-metrics-tls\") pod \"dns-default-whz62\" (UID: \"569a2ae0-d9c0-4f18-a27c-14167c5ccdcb\") " pod="openshift-dns/dns-default-whz62"
Oct 31 03:07:28.352040 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.352027    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fdf1fa41-a03b-4a64-ba80-4c7a2825e259-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-brs5l\" (UID: \"fdf1fa41-a03b-4a64-ba80-4c7a2825e259\") " pod="openshift-multus/multus-additional-cni-plugins-brs5l"
Oct 31 03:07:28.355274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.355255    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cda5dcc5-e5c6-4e0a-9acb-6191475f552c-ovnkube-script-lib\") pod \"ovnkube-node-xm5zs\" (UID: \"cda5dcc5-e5c6-4e0a-9acb-6191475f552c\") " pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:28.388352 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.388324    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 03:07:28.388352 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.388339    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 03:07:28.388571 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.388363    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 03:07:28.388571 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.388462    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 03:07:28.394756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.394734    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c48f7097-d8ba-4a86-9fe7-d56739a332f3-installation-pull-secrets\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.394988 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.394971    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c48f7097-d8ba-4a86-9fe7-d56739a332f3-trusted-ca\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.397034 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.397013    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f714c9b3-ccf2-4dab-9220-321c5b4258d0-original-pull-secret\") pod \"global-pull-secret-syncer-ndfgt\" (UID: \"f714c9b3-ccf2-4dab-9220-321c5b4258d0\") " pod="kube-system/global-pull-secret-syncer-ndfgt"
Oct 31 03:07:28.398070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.398056    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c48f7097-d8ba-4a86-9fe7-d56739a332f3-registry-tls\") pod \"image-registry-ccdffd8bb-vmv6b\" (UID: \"c48f7097-d8ba-4a86-9fe7-d56739a332f3\") " pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:28.493998 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:28.493970    4900 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:07:29.439416 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:29.439386    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-whz62"
Oct 31 03:07:29.772576 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:29.772548    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-ccdffd8bb-vmv6b"
Oct 31 03:07:30.257973 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:30.257934    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:30.452244 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:30.452215    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:30.477332 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:30.477301    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:30.727754 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:30.727665    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:30.956920 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:30.956886    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:31.230522 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.230492    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:31.453653 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.453622    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:31.673028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.672991    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:31.746529 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.746499    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wwhp2"]
Oct 31 03:07:31.773259 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.773226    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.853735 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.853696    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 03:07:31.868458 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.868432    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k74vl\""
Oct 31 03:07:31.868594 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.868462    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 03:07:31.873002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.872979    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/577fcc94-dce8-4841-bf6a-7102e073b362-sys\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.873113 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.873008    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.873113 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.873034    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-wtmp\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.873113 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.873052    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-tls\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.873113 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.873102    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-accelerators-collector-config\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.873246 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.873206    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/577fcc94-dce8-4841-bf6a-7102e073b362-metrics-client-ca\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.873246 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.873228    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/577fcc94-dce8-4841-bf6a-7102e073b362-root\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.873318 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.873244    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz94s\" (UniqueName: \"kubernetes.io/projected/577fcc94-dce8-4841-bf6a-7102e073b362-kube-api-access-rz94s\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.873318 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.873264    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-textfile\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.900192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.900148    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 03:07:31.900192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.900173    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 03:07:31.904346 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.904319    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:31.914860 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.914836    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 03:07:31.914961 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.914877    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 03:07:31.974216 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974134    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-textfile\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.974216 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974175    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/577fcc94-dce8-4841-bf6a-7102e073b362-sys\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.974216 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974198    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.974471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974228    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-wtmp\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.974471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974244    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-tls\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.974471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974251    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/577fcc94-dce8-4841-bf6a-7102e073b362-sys\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.974471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974259    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-accelerators-collector-config\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.974471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974317    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/577fcc94-dce8-4841-bf6a-7102e073b362-metrics-client-ca\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.974471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974356    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/577fcc94-dce8-4841-bf6a-7102e073b362-root\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.974471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974383    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz94s\" (UniqueName: \"kubernetes.io/projected/577fcc94-dce8-4841-bf6a-7102e073b362-kube-api-access-rz94s\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.974471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974408    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-wtmp\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.974756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974477    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/577fcc94-dce8-4841-bf6a-7102e073b362-root\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.974756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974539    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-textfile\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.974756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974627    4900 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 03:07:31.974970 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974951    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-accelerators-collector-config\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.975027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.974999    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/577fcc94-dce8-4841-bf6a-7102e073b362-metrics-client-ca\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.976756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.976735    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-tls\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:31.976860 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:31.976762    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/577fcc94-dce8-4841-bf6a-7102e073b362-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:32.031335 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:32.031298    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz94s\" (UniqueName: \"kubernetes.io/projected/577fcc94-dce8-4841-bf6a-7102e073b362-kube-api-access-rz94s\") pod \"node-exporter-wwhp2\" (UID: \"577fcc94-dce8-4841-bf6a-7102e073b362\") " pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:32.124903 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:32.124868    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wwhp2"
Oct 31 03:07:32.131245 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:32.131219    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:32.151481 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:32.151453    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577fcc94_dce8_4841_bf6a_7102e073b362.slice/crio-30692c3e96c82e5cafc9cd04cf1de11a2653ee7ea3cb021b01339221b8b55fc8 WatchSource:0}: Error finding container 30692c3e96c82e5cafc9cd04cf1de11a2653ee7ea3cb021b01339221b8b55fc8: Status 404 returned error can't find the container with id 30692c3e96c82e5cafc9cd04cf1de11a2653ee7ea3cb021b01339221b8b55fc8
Oct 31 03:07:32.203560 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:32.203525    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8z98n" event={"ID":"fdc104e8-81dd-4bef-8dee-e28bf5c7a10d","Type":"ContainerStarted","Data":"0077d402fc67b28d09089c69e388ee85c0decc91b805fc7cc18b3c42153a4e6c"}
Oct 31 03:07:32.204645 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:32.204615    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wwhp2" event={"ID":"577fcc94-dce8-4841-bf6a-7102e073b362","Type":"ContainerStarted","Data":"30692c3e96c82e5cafc9cd04cf1de11a2653ee7ea3cb021b01339221b8b55fc8"}
Oct 31 03:07:32.351795 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:32.351762    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:32.563676 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:32.563643    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:33.142299 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:33.142267    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:33.375462 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:33.375419    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:33.611495 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:33.611454    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:34.048344 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.048320    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:07:34.050807 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.050775    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.058150 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.058129    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mmv7s"
Oct 31 03:07:34.066221 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.066198    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:07:34.074990 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.074966    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:34.111583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.111553    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 03:07:34.111583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.111574    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 03:07:34.111807 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.111553    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 03:07:34.111807 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.111643    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 03:07:34.111807 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.111703    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 03:07:34.111807 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.111745    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9qqnx\""
Oct 31 03:07:34.111807 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.111769    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 03:07:34.111807 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.111779    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 03:07:34.111807 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.111806    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 03:07:34.190676 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.190590    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13cce5e-6c03-4d1b-9295-0a15df7403d5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.190676 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.190629    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a13cce5e-6c03-4d1b-9295-0a15df7403d5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.190676 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.190652    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-web-config\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.190976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.190842    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.190976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.190886    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nftlp\" (UniqueName: \"kubernetes.io/projected/a13cce5e-6c03-4d1b-9295-0a15df7403d5-kube-api-access-nftlp\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.190976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.190918    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-config-volume\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.190976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.190937    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a13cce5e-6c03-4d1b-9295-0a15df7403d5-config-out\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.190976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.190953    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a13cce5e-6c03-4d1b-9295-0a15df7403d5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.191139 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.191054    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.191139 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.191083    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.191139 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.191108    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.191247 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.191159    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13cce5e-6c03-4d1b-9295-0a15df7403d5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.191247 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.191177    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.191827 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.191808    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 03:07:34.211654 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.211623    4900 generic.go:356] "Generic (PLEG): container finished" podID="577fcc94-dce8-4841-bf6a-7102e073b362" containerID="6bd283f0215041f676c8958af49955621bf92e4879182fabac023eb78c33916a" exitCode=0
Oct 31 03:07:34.211843 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.211667    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wwhp2" event={"ID":"577fcc94-dce8-4841-bf6a-7102e073b362","Type":"ContainerDied","Data":"6bd283f0215041f676c8958af49955621bf92e4879182fabac023eb78c33916a"}
Oct 31 03:07:34.289519 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.289493    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5cd5fd947f-966d4"]
Oct 31 03:07:34.291922 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.291899    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13cce5e-6c03-4d1b-9295-0a15df7403d5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.292035 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.291939    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.292035 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.291990    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13cce5e-6c03-4d1b-9295-0a15df7403d5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.292141 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.292049    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a13cce5e-6c03-4d1b-9295-0a15df7403d5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.292141 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.292082    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-web-config\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.292141 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.292123    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.292291 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.292149    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nftlp\" (UniqueName: \"kubernetes.io/projected/a13cce5e-6c03-4d1b-9295-0a15df7403d5-kube-api-access-nftlp\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.292291 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.292204    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-config-volume\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.292291 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.292227    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a13cce5e-6c03-4d1b-9295-0a15df7403d5-config-out\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.292291 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.292251    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a13cce5e-6c03-4d1b-9295-0a15df7403d5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.292291 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.292284    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.292520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.292314    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.292520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.292366    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.293181 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.292862    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13cce5e-6c03-4d1b-9295-0a15df7403d5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.293181 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.292901    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13cce5e-6c03-4d1b-9295-0a15df7403d5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.293181 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.292943    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.293563 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.293537    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a13cce5e-6c03-4d1b-9295-0a15df7403d5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.294923 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.294890    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a13cce5e-6c03-4d1b-9295-0a15df7403d5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.295353 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.295328    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.295426 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.295397    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.295426 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.295402    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.296363 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.296342    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.296434 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.296374    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a13cce5e-6c03-4d1b-9295-0a15df7403d5-config-out\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.296471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.296447    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.296504 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.296491    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-web-config\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.297678 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.297661    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-config-volume\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.344399 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.344375    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 03:07:34.344748 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.344727    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vz8qq\""
Oct 31 03:07:34.344852 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.344756    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 03:07:34.344852 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.344767    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-f85b3th4puk31\""
Oct 31 03:07:34.344852 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.344764    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 03:07:34.345559 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.345542    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 03:07:34.345832 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.345809    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 03:07:34.364113 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.364089    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nftlp\" (UniqueName: \"kubernetes.io/projected/a13cce5e-6c03-4d1b-9295-0a15df7403d5-kube-api-access-nftlp\") pod \"alertmanager-main-0\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.380405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.380369    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5cd5fd947f-966d4"]
Oct 31 03:07:34.392893 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.392858    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b29ff389-bc68-4df1-86e3-0ea1c48b4182-tmp\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.393029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.392902    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.393029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.392927    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.393108 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.393018    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29ff389-bc68-4df1-86e3-0ea1c48b4182-metrics-client-ca\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.393108 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.393073    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-tls\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.393108 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.393095    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxtbs\" (UniqueName: \"kubernetes.io/projected/b29ff389-bc68-4df1-86e3-0ea1c48b4182-kube-api-access-mxtbs\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.393210 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.393166    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-grpc-tls\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.393210 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.393191    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.393278 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.393219    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.405910 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.405880    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:07:34.494445 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.494414    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.494445 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.494448    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b29ff389-bc68-4df1-86e3-0ea1c48b4182-tmp\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.494673 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.494469    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.494673 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.494490    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.494673 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.494512    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29ff389-bc68-4df1-86e3-0ea1c48b4182-metrics-client-ca\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.494673 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.494529    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-tls\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.494673 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.494547    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxtbs\" (UniqueName: \"kubernetes.io/projected/b29ff389-bc68-4df1-86e3-0ea1c48b4182-kube-api-access-mxtbs\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.494673 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.494580    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-grpc-tls\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.494949 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.494828    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.494949 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.494900    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b29ff389-bc68-4df1-86e3-0ea1c48b4182-tmp\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.495599 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.495571    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b29ff389-bc68-4df1-86e3-0ea1c48b4182-metrics-client-ca\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.497104 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.497067    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.497276 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.497251    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.497462 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.497441    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.497670 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.497652    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-tls\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.497709 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.497652    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.497709 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.497686    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b29ff389-bc68-4df1-86e3-0ea1c48b4182-secret-grpc-tls\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.547603 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.547572    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxtbs\" (UniqueName: \"kubernetes.io/projected/b29ff389-bc68-4df1-86e3-0ea1c48b4182-kube-api-access-mxtbs\") pod \"thanos-querier-5cd5fd947f-966d4\" (UID: \"b29ff389-bc68-4df1-86e3-0ea1c48b4182\") " pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.649588 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.649548    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:34.821862 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.821832    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:07:34.824886 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:34.824853    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13cce5e_6c03_4d1b_9295_0a15df7403d5.slice/crio-f7a9433a676e0073e2b57f170d60180b8775265160f76e59a5a1c099d8b181ad WatchSource:0}: Error finding container f7a9433a676e0073e2b57f170d60180b8775265160f76e59a5a1c099d8b181ad: Status 404 returned error can't find the container with id f7a9433a676e0073e2b57f170d60180b8775265160f76e59a5a1c099d8b181ad
Oct 31 03:07:34.830288 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:34.830265    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:35.010912 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:35.010874    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5cd5fd947f-966d4"]
Oct 31 03:07:35.015595 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:35.015558    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29ff389_bc68_4df1_86e3_0ea1c48b4182.slice/crio-d1c534fd13a3773ad0eb7c21b9cae4d06144bbe61152ccc0eee964cc5a366627 WatchSource:0}: Error finding container d1c534fd13a3773ad0eb7c21b9cae4d06144bbe61152ccc0eee964cc5a366627: Status 404 returned error can't find the container with id d1c534fd13a3773ad0eb7c21b9cae4d06144bbe61152ccc0eee964cc5a366627
Oct 31 03:07:35.042920 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:35.042891    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:35.216951 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:35.216865    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wwhp2" event={"ID":"577fcc94-dce8-4841-bf6a-7102e073b362","Type":"ContainerStarted","Data":"0a38e5c4a9fa61878878e900441df45a07470ec037de787f688d3b34b6795cb8"}
Oct 31 03:07:35.216951 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:35.216930    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wwhp2" event={"ID":"577fcc94-dce8-4841-bf6a-7102e073b362","Type":"ContainerStarted","Data":"9e9223103cc2d4f8f4dc69986f669a950cc7ed18bdc54796b72448d7c4c0b488"}
Oct 31 03:07:35.218064 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:35.218039    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4" event={"ID":"b29ff389-bc68-4df1-86e3-0ea1c48b4182","Type":"ContainerStarted","Data":"d1c534fd13a3773ad0eb7c21b9cae4d06144bbe61152ccc0eee964cc5a366627"}
Oct 31 03:07:35.219092 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:35.219074    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerStarted","Data":"f7a9433a676e0073e2b57f170d60180b8775265160f76e59a5a1c099d8b181ad"}
Oct 31 03:07:35.326693 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:35.326629    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wwhp2" podStartSLOduration=2.512208122 podStartE2EDuration="4.326609691s" totalImagesPullingTime="1.747702279s" totalInitContainerRuntime="66.69929ms" isStatefulPod=true podCreationTimestamp="2026-10-31 03:07:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:35.325903804 +0000 UTC m=+8.292237150" watchObservedRunningTime="2026-10-31 03:07:35.326609691 +0000 UTC m=+8.292943034"
Oct 31 03:07:35.820697 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:35.820667    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:35.821188 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:35.820953    4900 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:07:35.875753 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:35.875721    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:36.133652 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:36.133570    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:36.343836 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:36.343758    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:36.552493 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:36.552460    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:36.616067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:36.616035    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p5zp9_8b5e3f9c-61e6-4468-9891-afdeb408caf7/serve-healthcheck-canary/0.log"
Oct 31 03:07:36.775846 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:36.775815    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:36.984841 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:07:36.984806    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:07:36.985446 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:36.985386    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:37.234384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.234348    4900 generic.go:356] "Generic (PLEG): container finished" podID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerID="768b07aff70259cd2e7ce1f075057ee5b9267b4a7f31989f405ffad39a97c22b" exitCode=0
Oct 31 03:07:37.234550 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.234426    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerDied","Data":"768b07aff70259cd2e7ce1f075057ee5b9267b4a7f31989f405ffad39a97c22b"}
Oct 31 03:07:37.419379 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.419328    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:37.491491 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.491406    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-f9c76d5c-7fddq"]
Oct 31 03:07:37.493727 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.493701    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.542473 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.542438    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 03:07:37.542658 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.542633    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 03:07:37.542658 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.542634    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 03:07:37.542763 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.542641    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 03:07:37.542821 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.542804    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-282v6\""
Oct 31 03:07:37.542993 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.542977    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 03:07:37.589219 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.589198    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 03:07:37.590078 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.590059    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f9c76d5c-7fddq"]
Oct 31 03:07:37.620923 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.620902    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-federate-client-tls\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.621035 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.620946    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-serving-certs-ca-bundle\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.621035 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.621023    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.621131 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.621087    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.621131 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.621111    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfz6r\" (UniqueName: \"kubernetes.io/projected/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-kube-api-access-kfz6r\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.621232 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.621168    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-secret-telemeter-client\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.621232 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.621196    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-metrics-client-ca\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.621318 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.621282    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-telemeter-client-tls\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.638635 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.638608    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:37.721722 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.721700    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-federate-client-tls\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.721843 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.721738    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-serving-certs-ca-bundle\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.721929 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.721903    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.721984 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.721968    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.722019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.721998    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfz6r\" (UniqueName: \"kubernetes.io/projected/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-kube-api-access-kfz6r\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.722067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.722030    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-secret-telemeter-client\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.722067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.722057    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-metrics-client-ca\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.722171 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.722132    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-telemeter-client-tls\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.722447 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.722415    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-serving-certs-ca-bundle\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.723027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.723000    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.723128 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.723026    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-metrics-client-ca\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.724317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.724236    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-federate-client-tls\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.724317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.724284    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.724479 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.724460    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-secret-telemeter-client\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.725077 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.725057    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-telemeter-client-tls\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.775549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.775483    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfz6r\" (UniqueName: \"kubernetes.io/projected/7789c29a-d4b9-41b9-8a8a-6e48255f6e7d-kube-api-access-kfz6r\") pod \"telemeter-client-f9c76d5c-7fddq\" (UID: \"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d\") " pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:37.810078 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:37.810047    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq"
Oct 31 03:07:38.242679 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:38.242643    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4" event={"ID":"b29ff389-bc68-4df1-86e3-0ea1c48b4182","Type":"ContainerStarted","Data":"dfadc998c235c458a44eb1b7b95dd67c76e540e980380c2cfbfb29bdf018e57c"}
Oct 31 03:07:38.243081 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:38.242710    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4" event={"ID":"b29ff389-bc68-4df1-86e3-0ea1c48b4182","Type":"ContainerStarted","Data":"591e88b75ef58fe8fbc92271df458f4a94e24ba73567ddcf77270d3a00a586a6"}
Oct 31 03:07:38.243081 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:38.242736    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4" event={"ID":"b29ff389-bc68-4df1-86e3-0ea1c48b4182","Type":"ContainerStarted","Data":"5fae362970a9237b4892f5fb2bd9109d87e9a2decf9e234e9f35e722e127d8c8"}
Oct 31 03:07:38.263141 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:38.263109    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f9c76d5c-7fddq"]
Oct 31 03:07:38.299176 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:38.299150    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7789c29a_d4b9_41b9_8a8a_6e48255f6e7d.slice/crio-fc81cf307301301a51306790e1ce3ac0c4739a4fcdefeb731b14eda36d91eb0f WatchSource:0}: Error finding container fc81cf307301301a51306790e1ce3ac0c4739a4fcdefeb731b14eda36d91eb0f: Status 404 returned error can't find the container with id fc81cf307301301a51306790e1ce3ac0c4739a4fcdefeb731b14eda36d91eb0f
Oct 31 03:07:39.248494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:39.248401    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq" event={"ID":"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d","Type":"ContainerStarted","Data":"fc81cf307301301a51306790e1ce3ac0c4739a4fcdefeb731b14eda36d91eb0f"}
Oct 31 03:07:40.256095 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:40.256042    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4" event={"ID":"b29ff389-bc68-4df1-86e3-0ea1c48b4182","Type":"ContainerStarted","Data":"cf37b27f11ed813831e39c0a88c143259e649135459b5c70a4a0eb5badd4c633"}
Oct 31 03:07:40.256547 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:40.256173    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4" event={"ID":"b29ff389-bc68-4df1-86e3-0ea1c48b4182","Type":"ContainerStarted","Data":"3cb1ab7017c2db051ace26d4da2d8d9f6b45bc7c8f4ce592a5e2db53c34a7eae"}
Oct 31 03:07:40.256547 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:40.256217    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4" event={"ID":"b29ff389-bc68-4df1-86e3-0ea1c48b4182","Type":"ContainerStarted","Data":"24b28a03ddef46c8d752e2165626f2680db304fbc736a11ea4e5057b657602ad"}
Oct 31 03:07:40.256547 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:40.256265    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:40.259890 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:40.259863    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerStarted","Data":"3d96a99a3cc7f1055bdc21563ec00e1b10175b8d1f15f7820e71d7aeaf0860a1"}
Oct 31 03:07:40.260052 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:40.259932    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerStarted","Data":"ad6258e306c5a86fa72a9a153fa7ce3351ca345db868933b82dc064de55a2d38"}
Oct 31 03:07:40.260052 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:40.259979    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerStarted","Data":"25ed00a931bc33736f7c56db73f37c516a4c1f764183393767133165974e5ddf"}
Oct 31 03:07:40.260052 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:40.260016    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerStarted","Data":"026be83d339317d08bb56c60d3e72fd877483f765528e08588cbd7d2ac07aa57"}
Oct 31 03:07:40.260052 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:40.260046    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerStarted","Data":"49bd5553786e98fe97c71f129dcc82a1f72583ca3519098902c87623afcd2bc6"}
Oct 31 03:07:40.260234 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:40.260075    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerStarted","Data":"305e3c971588c09c458dfeff3019f10822827a7711005238f2fc820ac8f910c7"}
Oct 31 03:07:40.405425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:40.405365    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4" podStartSLOduration=2.321085308 podStartE2EDuration="6.40534863s" totalImagesPullingTime="4.084263322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:40.405315262 +0000 UTC m=+13.371648608" watchObservedRunningTime="2026-10-31 03:07:40.40534863 +0000 UTC m=+13.371681972"
Oct 31 03:07:40.521611 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:40.521521    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.243219811 podStartE2EDuration="7.521506036s" totalImagesPullingTime="4.262089474s" totalInitContainerRuntime="16.196751ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:40.520754031 +0000 UTC m=+13.487087374" watchObservedRunningTime="2026-10-31 03:07:40.521506036 +0000 UTC m=+13.487839380"
Oct 31 03:07:40.697388 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:40.697361    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:41.265296 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:41.265261    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq" event={"ID":"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d","Type":"ContainerStarted","Data":"8e328313f1d8b18310d23e71ddd4781c3088cd2f68e2a6cbd8822199fb298fd5"}
Oct 31 03:07:41.265814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:41.265322    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq" event={"ID":"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d","Type":"ContainerStarted","Data":"5d23add6e4d58127101dbab2cf8b937581f10c52e46131f91b61fd7e969c99e9"}
Oct 31 03:07:41.265814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:41.265350    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq" event={"ID":"7789c29a-d4b9-41b9-8a8a-6e48255f6e7d","Type":"ContainerStarted","Data":"093546ceaa2d7c7d80d632417e6c8ab93e6a66ff4dabbb66131f2789b20aa0fa"}
Oct 31 03:07:41.379340 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:41.379291    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-f9c76d5c-7fddq" podStartSLOduration=2.102183142 podStartE2EDuration="4.379277417s" totalImagesPullingTime="2.277094275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:41.378629751 +0000 UTC m=+14.344963095" watchObservedRunningTime="2026-10-31 03:07:41.379277417 +0000 UTC m=+14.345610761"
Oct 31 03:07:41.709443 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:41.709342    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:41.937348 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:41.937300    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:42.426806 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:42.426763    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c7b69"
Oct 31 03:07:44.249049 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:44.249007    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:44.474704 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:44.474661    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:45.121931 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.121897    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f74698894-dkrft"]
Oct 31 03:07:45.127046 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.127023    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.175958 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.175930    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 03:07:45.176094 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.175971    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 03:07:45.176094 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.175973    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 03:07:45.222203 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.222132    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 03:07:45.222203 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.222132    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 03:07:45.281617 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.281584    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-service-ca\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.281617 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.281615    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-oauth-serving-cert\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.282124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.281634    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntz4r\" (UniqueName: \"kubernetes.io/projected/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-kube-api-access-ntz4r\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.282124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.281658    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-tmp\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.282124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.281831    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-serving-cert\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.282124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.281892    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-trusted-ca-bundle\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.282124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.281963    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-config\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.282124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.282001    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-oauth-config\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.315655 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.315623    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 03:07:45.315803 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.315689    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 03:07:45.315803 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.315689    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fb2gc\""
Oct 31 03:07:45.315803 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.315711    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 03:07:45.316576 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.316559    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f74698894-dkrft"]
Oct 31 03:07:45.383276 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.383204    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntz4r\" (UniqueName: \"kubernetes.io/projected/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-kube-api-access-ntz4r\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.383276 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.383251    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-tmp\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.383480 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.383294    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-serving-cert\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.383480 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.383340    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-trusted-ca-bundle\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.383480 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.383461    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-config\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.383634 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.383527    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-oauth-config\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.383634 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.383583    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-service-ca\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.383634 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.383611    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-oauth-serving-cert\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.383815 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.383717    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-tmp\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.384162 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.384139    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-config\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.384291 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.384205    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-trusted-ca-bundle\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.384371 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.384290    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-service-ca\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.384404 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.384387    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-oauth-serving-cert\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.385914 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.385897    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-serving-cert\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.385996 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.385916    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-oauth-config\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.440968 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.440942    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntz4r\" (UniqueName: \"kubernetes.io/projected/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-kube-api-access-ntz4r\") pod \"console-7f74698894-dkrft\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") " pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:45.738953 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:45.738917    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:46.105465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:46.105389    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f74698894-dkrft"]
Oct 31 03:07:46.133271 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:07:46.133245    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd722c6fb_13d0_47fe_aee9_8b1fb73fa3d1.slice/crio-dc6b916928d2c34c1b0382a6052caf80e565a9e9b557ab3168b855cbe7b7f14f WatchSource:0}: Error finding container dc6b916928d2c34c1b0382a6052caf80e565a9e9b557ab3168b855cbe7b7f14f: Status 404 returned error can't find the container with id dc6b916928d2c34c1b0382a6052caf80e565a9e9b557ab3168b855cbe7b7f14f
Oct 31 03:07:46.272197 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:46.272169    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5cd5fd947f-966d4"
Oct 31 03:07:46.283945 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:46.283912    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f74698894-dkrft" event={"ID":"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1","Type":"ContainerStarted","Data":"dc6b916928d2c34c1b0382a6052caf80e565a9e9b557ab3168b855cbe7b7f14f"}
Oct 31 03:07:47.182042 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:47.182010    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:47.389551 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:47.389513    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:50.300082 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:50.300046    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f74698894-dkrft" event={"ID":"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1","Type":"ContainerStarted","Data":"d0833e2ebd8b8d0a61654a3c8c8bdf62d95a84438aec6b5a43d3488ee5069c07"}
Oct 31 03:07:50.409779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:50.409730    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7f74698894-dkrft" podStartSLOduration=1.631524528 podStartE2EDuration="5.409715996s" totalImagesPullingTime="3.778191468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:07:50.408767496 +0000 UTC m=+23.375100841" watchObservedRunningTime="2026-10-31 03:07:50.409715996 +0000 UTC m=+23.376049340"
Oct 31 03:07:52.406243 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:52.406159    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:52.687177 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:52.687104    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:53.806736 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:53.806704    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xm5zs"
Oct 31 03:07:54.693628 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:54.693594    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:54.904939 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:54.904906    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:55.109837 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:55.109807    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:07:55.739719 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:55.739684    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:55.739932 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:55.739749    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:55.744546 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:55.744522    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:07:56.337044 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:07:56.337019    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:08:06.983706 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:08:06.983670    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:08:07.406219 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:07.406144    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:07.633999 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:07.633948    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:12.885140 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:12.885102    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:13.092644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:13.092612    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:20.559121 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:20.559087    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:20.810263 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:20.810182    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:22.275641 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.275611    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b4dcb7b66-t7rrk"]
Oct 31 03:08:22.278767 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.278750    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.332837 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.332773    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b4dcb7b66-t7rrk"]
Oct 31 03:08:22.401064 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.401039    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgn87\" (UniqueName: \"kubernetes.io/projected/d3eeb220-3c3f-405b-bfba-a3652d00042a-kube-api-access-rgn87\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.401184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.401075    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-3c3f-405b-bfba-a3652d00042a-oauth-serving-cert\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.401184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.401093    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-3c3f-405b-bfba-a3652d00042a-service-ca\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.401184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.401162    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-3c3f-405b-bfba-a3652d00042a-trusted-ca-bundle\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.401285 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.401260    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3eeb220-3c3f-405b-bfba-a3652d00042a-console-oauth-config\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.401318 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.401308    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-3c3f-405b-bfba-a3652d00042a-console-config\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.401388 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.401370    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3eeb220-3c3f-405b-bfba-a3652d00042a-console-serving-cert\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.401420 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.401400    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3eeb220-3c3f-405b-bfba-a3652d00042a-tmp\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.408360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.408335    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:22.502744 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.502716    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgn87\" (UniqueName: \"kubernetes.io/projected/d3eeb220-3c3f-405b-bfba-a3652d00042a-kube-api-access-rgn87\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.502897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.502749    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-3c3f-405b-bfba-a3652d00042a-oauth-serving-cert\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.502897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.502770    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-3c3f-405b-bfba-a3652d00042a-service-ca\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.502981 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.502914    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-3c3f-405b-bfba-a3652d00042a-trusted-ca-bundle\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.502981 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.502952    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3eeb220-3c3f-405b-bfba-a3652d00042a-console-oauth-config\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.503062 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.503035    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-3c3f-405b-bfba-a3652d00042a-console-config\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.503112 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.503084    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3eeb220-3c3f-405b-bfba-a3652d00042a-console-serving-cert\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.503160 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.503114    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3eeb220-3c3f-405b-bfba-a3652d00042a-tmp\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.503474 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.503443    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-3c3f-405b-bfba-a3652d00042a-service-ca\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.503580 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.503524    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3eeb220-3c3f-405b-bfba-a3652d00042a-tmp\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.503580 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.503532    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-3c3f-405b-bfba-a3652d00042a-oauth-serving-cert\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.503751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.503728    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-3c3f-405b-bfba-a3652d00042a-console-config\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.503825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.503802    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3eeb220-3c3f-405b-bfba-a3652d00042a-trusted-ca-bundle\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.505421 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.505394    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3eeb220-3c3f-405b-bfba-a3652d00042a-console-serving-cert\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.505516 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.505447    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3eeb220-3c3f-405b-bfba-a3652d00042a-console-oauth-config\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.556381 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.556319    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgn87\" (UniqueName: \"kubernetes.io/projected/d3eeb220-3c3f-405b-bfba-a3652d00042a-kube-api-access-rgn87\") pod \"console-7b4dcb7b66-t7rrk\" (UID: \"d3eeb220-3c3f-405b-bfba-a3652d00042a\") " pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.588768 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.588741    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:22.612321 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.612292    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:22.951072 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.950994    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b4dcb7b66-t7rrk"]
Oct 31 03:08:22.953819 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:08:22.953776    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3eeb220_3c3f_405b_bfba_a3652d00042a.slice/crio-6943acb6fbeeec99edb0da591e920e89ff7a33ceb7ffa66ef98f3fdcbf6702f3 WatchSource:0}: Error finding container 6943acb6fbeeec99edb0da591e920e89ff7a33ceb7ffa66ef98f3fdcbf6702f3: Status 404 returned error can't find the container with id 6943acb6fbeeec99edb0da591e920e89ff7a33ceb7ffa66ef98f3fdcbf6702f3
Oct 31 03:08:22.966992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:22.966967    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:23.179240 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:23.179207    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:23.421638 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:23.421602    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b4dcb7b66-t7rrk" event={"ID":"d3eeb220-3c3f-405b-bfba-a3652d00042a","Type":"ContainerStarted","Data":"45aba0f973768f5bc88f29a3e2c1bf610a80688fbb81a56d327ffd268422aa4c"}
Oct 31 03:08:23.422062 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:23.421654    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b4dcb7b66-t7rrk" event={"ID":"d3eeb220-3c3f-405b-bfba-a3652d00042a","Type":"ContainerStarted","Data":"6943acb6fbeeec99edb0da591e920e89ff7a33ceb7ffa66ef98f3fdcbf6702f3"}
Oct 31 03:08:23.529585 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:23.529539    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7b4dcb7b66-t7rrk" podStartSLOduration=1.529526594 podStartE2EDuration="1.529526594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:23.529273275 +0000 UTC m=+56.495606619" watchObservedRunningTime="2026-10-31 03:08:23.529526594 +0000 UTC m=+56.495859929"
Oct 31 03:08:28.089849 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.089812    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:08:28.090272 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.090247    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="kube-rbac-proxy-metric" containerID="cri-o://ad6258e306c5a86fa72a9a153fa7ce3351ca345db868933b82dc064de55a2d38" gracePeriod=120
Oct 31 03:08:28.090334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.090247    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="alertmanager" containerID="cri-o://305e3c971588c09c458dfeff3019f10822827a7711005238f2fc820ac8f910c7" gracePeriod=120
Oct 31 03:08:28.090334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.090274    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="kube-rbac-proxy-web" containerID="cri-o://026be83d339317d08bb56c60d3e72fd877483f765528e08588cbd7d2ac07aa57" gracePeriod=120
Oct 31 03:08:28.090334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.090311    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="prom-label-proxy" containerID="cri-o://3d96a99a3cc7f1055bdc21563ec00e1b10175b8d1f15f7820e71d7aeaf0860a1" gracePeriod=120
Oct 31 03:08:28.090483 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.090353    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="config-reloader" containerID="cri-o://49bd5553786e98fe97c71f129dcc82a1f72583ca3519098902c87623afcd2bc6" gracePeriod=120
Oct 31 03:08:28.090483 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.090284    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="kube-rbac-proxy" containerID="cri-o://25ed00a931bc33736f7c56db73f37c516a4c1f764183393767133165974e5ddf" gracePeriod=120
Oct 31 03:08:28.440531 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.440444    4900 generic.go:356] "Generic (PLEG): container finished" podID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerID="3d96a99a3cc7f1055bdc21563ec00e1b10175b8d1f15f7820e71d7aeaf0860a1" exitCode=0
Oct 31 03:08:28.440531 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.440471    4900 generic.go:356] "Generic (PLEG): container finished" podID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerID="ad6258e306c5a86fa72a9a153fa7ce3351ca345db868933b82dc064de55a2d38" exitCode=0
Oct 31 03:08:28.440531 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.440479    4900 generic.go:356] "Generic (PLEG): container finished" podID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerID="25ed00a931bc33736f7c56db73f37c516a4c1f764183393767133165974e5ddf" exitCode=0
Oct 31 03:08:28.440531 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.440487    4900 generic.go:356] "Generic (PLEG): container finished" podID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerID="49bd5553786e98fe97c71f129dcc82a1f72583ca3519098902c87623afcd2bc6" exitCode=0
Oct 31 03:08:28.440531 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.440495    4900 generic.go:356] "Generic (PLEG): container finished" podID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerID="305e3c971588c09c458dfeff3019f10822827a7711005238f2fc820ac8f910c7" exitCode=0
Oct 31 03:08:28.440531 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.440515    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerDied","Data":"3d96a99a3cc7f1055bdc21563ec00e1b10175b8d1f15f7820e71d7aeaf0860a1"}
Oct 31 03:08:28.440977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.440609    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerDied","Data":"ad6258e306c5a86fa72a9a153fa7ce3351ca345db868933b82dc064de55a2d38"}
Oct 31 03:08:28.440977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.440636    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerDied","Data":"25ed00a931bc33736f7c56db73f37c516a4c1f764183393767133165974e5ddf"}
Oct 31 03:08:28.440977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.440660    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerDied","Data":"49bd5553786e98fe97c71f129dcc82a1f72583ca3519098902c87623afcd2bc6"}
Oct 31 03:08:28.440977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:28.440682    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerDied","Data":"305e3c971588c09c458dfeff3019f10822827a7711005238f2fc820ac8f910c7"}
Oct 31 03:08:29.339324 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.339298    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:29.446617 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.446537    4900 generic.go:356] "Generic (PLEG): container finished" podID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerID="026be83d339317d08bb56c60d3e72fd877483f765528e08588cbd7d2ac07aa57" exitCode=0
Oct 31 03:08:29.446617 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.446582    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerDied","Data":"026be83d339317d08bb56c60d3e72fd877483f765528e08588cbd7d2ac07aa57"}
Oct 31 03:08:29.446810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.446621    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:29.446810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.446638    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a13cce5e-6c03-4d1b-9295-0a15df7403d5","Type":"ContainerDied","Data":"f7a9433a676e0073e2b57f170d60180b8775265160f76e59a5a1c099d8b181ad"}
Oct 31 03:08:29.446810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.446674    4900 scope.go:122] "RemoveContainer" containerID="3d96a99a3cc7f1055bdc21563ec00e1b10175b8d1f15f7820e71d7aeaf0860a1"
Oct 31 03:08:29.453961 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.453941    4900 scope.go:122] "RemoveContainer" containerID="ad6258e306c5a86fa72a9a153fa7ce3351ca345db868933b82dc064de55a2d38"
Oct 31 03:08:29.460557 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.460528    4900 scope.go:122] "RemoveContainer" containerID="25ed00a931bc33736f7c56db73f37c516a4c1f764183393767133165974e5ddf"
Oct 31 03:08:29.463592 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.463572    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-config-volume\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-config-volume\") pod \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") "
Oct 31 03:08:29.463683 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.463620    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13cce5e-6c03-4d1b-9295-0a15df7403d5-tls-assets\" (UniqueName: \"kubernetes.io/projected/a13cce5e-6c03-4d1b-9295-0a15df7403d5-tls-assets\") pod \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") "
Oct 31 03:08:29.463683 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.463657    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13cce5e-6c03-4d1b-9295-0a15df7403d5-kube-api-access-nftlp\" (UniqueName: \"kubernetes.io/projected/a13cce5e-6c03-4d1b-9295-0a15df7403d5-kube-api-access-nftlp\") pod \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") "
Oct 31 03:08:29.463844 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.463698    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-main-tls\") pod \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") "
Oct 31 03:08:29.463902 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.463840    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-web-config\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-web-config\") pod \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") "
Oct 31 03:08:29.463961 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.463903    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy-web\") pod \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") "
Oct 31 03:08:29.463961 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.463947    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13cce5e-6c03-4d1b-9295-0a15df7403d5-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a13cce5e-6c03-4d1b-9295-0a15df7403d5-alertmanager-main-db\") pod \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") "
Oct 31 03:08:29.464077 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.463977    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy\") pod \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") "
Oct 31 03:08:29.464077 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.464027    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13cce5e-6c03-4d1b-9295-0a15df7403d5-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13cce5e-6c03-4d1b-9295-0a15df7403d5-metrics-client-ca\") pod \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") "
Oct 31 03:08:29.464077 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.464052    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-cluster-tls-config\") pod \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") "
Oct 31 03:08:29.464225 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.464095    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13cce5e-6c03-4d1b-9295-0a15df7403d5-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13cce5e-6c03-4d1b-9295-0a15df7403d5-alertmanager-trusted-ca-bundle\") pod \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") "
Oct 31 03:08:29.464225 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.464127    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13cce5e-6c03-4d1b-9295-0a15df7403d5-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a13cce5e-6c03-4d1b-9295-0a15df7403d5-config-out\") pod \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") "
Oct 31 03:08:29.464225 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.464168    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\" (UID: \"a13cce5e-6c03-4d1b-9295-0a15df7403d5\") "
Oct 31 03:08:29.464399 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.464363    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13cce5e-6c03-4d1b-9295-0a15df7403d5-alertmanager-main-db" pod "a13cce5e-6c03-4d1b-9295-0a15df7403d5" (UID: "a13cce5e-6c03-4d1b-9295-0a15df7403d5"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:08:29.465060 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.464781    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13cce5e-6c03-4d1b-9295-0a15df7403d5-alertmanager-trusted-ca-bundle" pod "a13cce5e-6c03-4d1b-9295-0a15df7403d5" (UID: "a13cce5e-6c03-4d1b-9295-0a15df7403d5"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:29.465805 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.465750    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13cce5e-6c03-4d1b-9295-0a15df7403d5-metrics-client-ca" pod "a13cce5e-6c03-4d1b-9295-0a15df7403d5" (UID: "a13cce5e-6c03-4d1b-9295-0a15df7403d5"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:29.468248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.468147    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-config-volume" pod "a13cce5e-6c03-4d1b-9295-0a15df7403d5" (UID: "a13cce5e-6c03-4d1b-9295-0a15df7403d5"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:29.468248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.468207    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-main-tls" pod "a13cce5e-6c03-4d1b-9295-0a15df7403d5" (UID: "a13cce5e-6c03-4d1b-9295-0a15df7403d5"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:29.468248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.468221    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13cce5e-6c03-4d1b-9295-0a15df7403d5-tls-assets" pod "a13cce5e-6c03-4d1b-9295-0a15df7403d5" (UID: "a13cce5e-6c03-4d1b-9295-0a15df7403d5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:08:29.468248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.468234    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy-web" pod "a13cce5e-6c03-4d1b-9295-0a15df7403d5" (UID: "a13cce5e-6c03-4d1b-9295-0a15df7403d5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:29.468517 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.468223    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13cce5e-6c03-4d1b-9295-0a15df7403d5-config-out" pod "a13cce5e-6c03-4d1b-9295-0a15df7403d5" (UID: "a13cce5e-6c03-4d1b-9295-0a15df7403d5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:08:29.468517 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.468481    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy" pod "a13cce5e-6c03-4d1b-9295-0a15df7403d5" (UID: "a13cce5e-6c03-4d1b-9295-0a15df7403d5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:29.468888 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.468870    4900 scope.go:122] "RemoveContainer" containerID="026be83d339317d08bb56c60d3e72fd877483f765528e08588cbd7d2ac07aa57"
Oct 31 03:08:29.469049 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.469021    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13cce5e-6c03-4d1b-9295-0a15df7403d5-kube-api-access-nftlp" pod "a13cce5e-6c03-4d1b-9295-0a15df7403d5" (UID: "a13cce5e-6c03-4d1b-9295-0a15df7403d5"). InnerVolumeSpecName "kube-api-access-nftlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:08:29.470099 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.469817    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy-metric" pod "a13cce5e-6c03-4d1b-9295-0a15df7403d5" (UID: "a13cce5e-6c03-4d1b-9295-0a15df7403d5"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:29.472491 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.472462    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-cluster-tls-config" pod "a13cce5e-6c03-4d1b-9295-0a15df7403d5" (UID: "a13cce5e-6c03-4d1b-9295-0a15df7403d5"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:29.478432 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.478404    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-web-config" pod "a13cce5e-6c03-4d1b-9295-0a15df7403d5" (UID: "a13cce5e-6c03-4d1b-9295-0a15df7403d5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:29.484666 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.484647    4900 scope.go:122] "RemoveContainer" containerID="49bd5553786e98fe97c71f129dcc82a1f72583ca3519098902c87623afcd2bc6"
Oct 31 03:08:29.491422 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.491404    4900 scope.go:122] "RemoveContainer" containerID="305e3c971588c09c458dfeff3019f10822827a7711005238f2fc820ac8f910c7"
Oct 31 03:08:29.497890 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.497871    4900 scope.go:122] "RemoveContainer" containerID="768b07aff70259cd2e7ce1f075057ee5b9267b4a7f31989f405ffad39a97c22b"
Oct 31 03:08:29.504524 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.504501    4900 scope.go:122] "RemoveContainer" containerID="3d96a99a3cc7f1055bdc21563ec00e1b10175b8d1f15f7820e71d7aeaf0860a1"
Oct 31 03:08:29.504866 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:08:29.504842    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d96a99a3cc7f1055bdc21563ec00e1b10175b8d1f15f7820e71d7aeaf0860a1\": container with ID starting with 3d96a99a3cc7f1055bdc21563ec00e1b10175b8d1f15f7820e71d7aeaf0860a1 not found: ID does not exist" containerID="3d96a99a3cc7f1055bdc21563ec00e1b10175b8d1f15f7820e71d7aeaf0860a1"
Oct 31 03:08:29.504948 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.504875    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d96a99a3cc7f1055bdc21563ec00e1b10175b8d1f15f7820e71d7aeaf0860a1"} err="failed to get container status \"3d96a99a3cc7f1055bdc21563ec00e1b10175b8d1f15f7820e71d7aeaf0860a1\": rpc error: code = NotFound desc = could not find container \"3d96a99a3cc7f1055bdc21563ec00e1b10175b8d1f15f7820e71d7aeaf0860a1\": container with ID starting with 3d96a99a3cc7f1055bdc21563ec00e1b10175b8d1f15f7820e71d7aeaf0860a1 not found: ID does not exist"
Oct 31 03:08:29.504948 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.504924    4900 scope.go:122] "RemoveContainer" containerID="ad6258e306c5a86fa72a9a153fa7ce3351ca345db868933b82dc064de55a2d38"
Oct 31 03:08:29.505164 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:08:29.505148    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad6258e306c5a86fa72a9a153fa7ce3351ca345db868933b82dc064de55a2d38\": container with ID starting with ad6258e306c5a86fa72a9a153fa7ce3351ca345db868933b82dc064de55a2d38 not found: ID does not exist" containerID="ad6258e306c5a86fa72a9a153fa7ce3351ca345db868933b82dc064de55a2d38"
Oct 31 03:08:29.505239 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.505166    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad6258e306c5a86fa72a9a153fa7ce3351ca345db868933b82dc064de55a2d38"} err="failed to get container status \"ad6258e306c5a86fa72a9a153fa7ce3351ca345db868933b82dc064de55a2d38\": rpc error: code = NotFound desc = could not find container \"ad6258e306c5a86fa72a9a153fa7ce3351ca345db868933b82dc064de55a2d38\": container with ID starting with ad6258e306c5a86fa72a9a153fa7ce3351ca345db868933b82dc064de55a2d38 not found: ID does not exist"
Oct 31 03:08:29.505239 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.505178    4900 scope.go:122] "RemoveContainer" containerID="25ed00a931bc33736f7c56db73f37c516a4c1f764183393767133165974e5ddf"
Oct 31 03:08:29.505428 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:08:29.505406    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ed00a931bc33736f7c56db73f37c516a4c1f764183393767133165974e5ddf\": container with ID starting with 25ed00a931bc33736f7c56db73f37c516a4c1f764183393767133165974e5ddf not found: ID does not exist" containerID="25ed00a931bc33736f7c56db73f37c516a4c1f764183393767133165974e5ddf"
Oct 31 03:08:29.505477 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.505433    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ed00a931bc33736f7c56db73f37c516a4c1f764183393767133165974e5ddf"} err="failed to get container status \"25ed00a931bc33736f7c56db73f37c516a4c1f764183393767133165974e5ddf\": rpc error: code = NotFound desc = could not find container \"25ed00a931bc33736f7c56db73f37c516a4c1f764183393767133165974e5ddf\": container with ID starting with 25ed00a931bc33736f7c56db73f37c516a4c1f764183393767133165974e5ddf not found: ID does not exist"
Oct 31 03:08:29.505477 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.505450    4900 scope.go:122] "RemoveContainer" containerID="026be83d339317d08bb56c60d3e72fd877483f765528e08588cbd7d2ac07aa57"
Oct 31 03:08:29.505679 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:08:29.505659    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026be83d339317d08bb56c60d3e72fd877483f765528e08588cbd7d2ac07aa57\": container with ID starting with 026be83d339317d08bb56c60d3e72fd877483f765528e08588cbd7d2ac07aa57 not found: ID does not exist" containerID="026be83d339317d08bb56c60d3e72fd877483f765528e08588cbd7d2ac07aa57"
Oct 31 03:08:29.505723 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.505685    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026be83d339317d08bb56c60d3e72fd877483f765528e08588cbd7d2ac07aa57"} err="failed to get container status \"026be83d339317d08bb56c60d3e72fd877483f765528e08588cbd7d2ac07aa57\": rpc error: code = NotFound desc = could not find container \"026be83d339317d08bb56c60d3e72fd877483f765528e08588cbd7d2ac07aa57\": container with ID starting with 026be83d339317d08bb56c60d3e72fd877483f765528e08588cbd7d2ac07aa57 not found: ID does not exist"
Oct 31 03:08:29.505723 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.505701    4900 scope.go:122] "RemoveContainer" containerID="49bd5553786e98fe97c71f129dcc82a1f72583ca3519098902c87623afcd2bc6"
Oct 31 03:08:29.505938 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:08:29.505921    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49bd5553786e98fe97c71f129dcc82a1f72583ca3519098902c87623afcd2bc6\": container with ID starting with 49bd5553786e98fe97c71f129dcc82a1f72583ca3519098902c87623afcd2bc6 not found: ID does not exist" containerID="49bd5553786e98fe97c71f129dcc82a1f72583ca3519098902c87623afcd2bc6"
Oct 31 03:08:29.506009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.505946    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49bd5553786e98fe97c71f129dcc82a1f72583ca3519098902c87623afcd2bc6"} err="failed to get container status \"49bd5553786e98fe97c71f129dcc82a1f72583ca3519098902c87623afcd2bc6\": rpc error: code = NotFound desc = could not find container \"49bd5553786e98fe97c71f129dcc82a1f72583ca3519098902c87623afcd2bc6\": container with ID starting with 49bd5553786e98fe97c71f129dcc82a1f72583ca3519098902c87623afcd2bc6 not found: ID does not exist"
Oct 31 03:08:29.506009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.505967    4900 scope.go:122] "RemoveContainer" containerID="305e3c971588c09c458dfeff3019f10822827a7711005238f2fc820ac8f910c7"
Oct 31 03:08:29.506192 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:08:29.506177    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305e3c971588c09c458dfeff3019f10822827a7711005238f2fc820ac8f910c7\": container with ID starting with 305e3c971588c09c458dfeff3019f10822827a7711005238f2fc820ac8f910c7 not found: ID does not exist" containerID="305e3c971588c09c458dfeff3019f10822827a7711005238f2fc820ac8f910c7"
Oct 31 03:08:29.506232 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.506196    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305e3c971588c09c458dfeff3019f10822827a7711005238f2fc820ac8f910c7"} err="failed to get container status \"305e3c971588c09c458dfeff3019f10822827a7711005238f2fc820ac8f910c7\": rpc error: code = NotFound desc = could not find container \"305e3c971588c09c458dfeff3019f10822827a7711005238f2fc820ac8f910c7\": container with ID starting with 305e3c971588c09c458dfeff3019f10822827a7711005238f2fc820ac8f910c7 not found: ID does not exist"
Oct 31 03:08:29.506232 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.506208    4900 scope.go:122] "RemoveContainer" containerID="768b07aff70259cd2e7ce1f075057ee5b9267b4a7f31989f405ffad39a97c22b"
Oct 31 03:08:29.506398 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:08:29.506381    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768b07aff70259cd2e7ce1f075057ee5b9267b4a7f31989f405ffad39a97c22b\": container with ID starting with 768b07aff70259cd2e7ce1f075057ee5b9267b4a7f31989f405ffad39a97c22b not found: ID does not exist" containerID="768b07aff70259cd2e7ce1f075057ee5b9267b4a7f31989f405ffad39a97c22b"
Oct 31 03:08:29.506447 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.506404    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768b07aff70259cd2e7ce1f075057ee5b9267b4a7f31989f405ffad39a97c22b"} err="failed to get container status \"768b07aff70259cd2e7ce1f075057ee5b9267b4a7f31989f405ffad39a97c22b\": rpc error: code = NotFound desc = could not find container \"768b07aff70259cd2e7ce1f075057ee5b9267b4a7f31989f405ffad39a97c22b\": container with ID starting with 768b07aff70259cd2e7ce1f075057ee5b9267b4a7f31989f405ffad39a97c22b not found: ID does not exist"
Oct 31 03:08:29.565829 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.565801    4900 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a13cce5e-6c03-4d1b-9295-0a15df7403d5-config-out\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:29.565829 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.565825    4900 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:29.565829 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.565838    4900 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:29.566109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.565846    4900 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a13cce5e-6c03-4d1b-9295-0a15df7403d5-tls-assets\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:29.566109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.565856    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nftlp\" (UniqueName: \"kubernetes.io/projected/a13cce5e-6c03-4d1b-9295-0a15df7403d5-kube-api-access-nftlp\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:29.566109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.565865    4900 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-main-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:29.566109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.565874    4900 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-web-config\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:29.566109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.565882    4900 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:29.566109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.565893    4900 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a13cce5e-6c03-4d1b-9295-0a15df7403d5-alertmanager-main-db\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:29.566109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.565901    4900 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:29.566109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.565910    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a13cce5e-6c03-4d1b-9295-0a15df7403d5-metrics-client-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:29.566109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.565918    4900 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a13cce5e-6c03-4d1b-9295-0a15df7403d5-cluster-tls-config\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:29.566109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.565927    4900 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13cce5e-6c03-4d1b-9295-0a15df7403d5-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:29.860995 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.860957    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:08:29.907575 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.907541    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:08:29.908100 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908079    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:08:29.908340 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908328    4900 cpu_manager.go:414] "containerMap: removing container" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="config-reloader"
Oct 31 03:08:29.908379 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908341    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="config-reloader"
Oct 31 03:08:29.908379 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908355    4900 cpu_manager.go:414] "containerMap: removing container" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="alertmanager"
Oct 31 03:08:29.908379 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908359    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="alertmanager"
Oct 31 03:08:29.908379 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908373    4900 cpu_manager.go:414] "containerMap: removing container" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="prom-label-proxy"
Oct 31 03:08:29.908379 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908378    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="prom-label-proxy"
Oct 31 03:08:29.908520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908386    4900 cpu_manager.go:414] "containerMap: removing container" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="init-config-reloader"
Oct 31 03:08:29.908520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908391    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="init-config-reloader"
Oct 31 03:08:29.908520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908401    4900 cpu_manager.go:414] "containerMap: removing container" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="kube-rbac-proxy"
Oct 31 03:08:29.908520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908405    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="kube-rbac-proxy"
Oct 31 03:08:29.908520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908413    4900 cpu_manager.go:414] "containerMap: removing container" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="kube-rbac-proxy-metric"
Oct 31 03:08:29.908520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908418    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="kube-rbac-proxy-metric"
Oct 31 03:08:29.908520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908429    4900 cpu_manager.go:414] "containerMap: removing container" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="kube-rbac-proxy-web"
Oct 31 03:08:29.908520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908434    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="kube-rbac-proxy-web"
Oct 31 03:08:29.908520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908471    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="kube-rbac-proxy-metric"
Oct 31 03:08:29.908520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908478    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="kube-rbac-proxy-web"
Oct 31 03:08:29.908520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908484    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="config-reloader"
Oct 31 03:08:29.908520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908490    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="alertmanager"
Oct 31 03:08:29.908520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908496    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="prom-label-proxy"
Oct 31 03:08:29.908520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.908502    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" containerName="kube-rbac-proxy"
Oct 31 03:08:29.912273 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.912257    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:29.973915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.973173    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 03:08:29.973915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.973246    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 03:08:29.973915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.973317    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 03:08:29.973915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.973326    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9qqnx\""
Oct 31 03:08:29.973915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.973373    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 03:08:29.973915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.973424    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 03:08:29.973915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.973428    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 03:08:29.973915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.973490    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 03:08:29.973915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:29.973501    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 03:08:30.019483 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.019395    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 03:08:30.023451 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.023409    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:08:30.074132 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.074084    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.074132 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.074130    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/16dab515-6397-4259-bfff-7cb9e0bd83de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.074334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.074148    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-config-volume\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.074334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.074218    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.074334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.074247    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16dab515-6397-4259-bfff-7cb9e0bd83de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.074427 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.074356    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhhzx\" (UniqueName: \"kubernetes.io/projected/16dab515-6397-4259-bfff-7cb9e0bd83de-kube-api-access-rhhzx\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.074427 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.074377    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.074427 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.074408    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16dab515-6397-4259-bfff-7cb9e0bd83de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.074517 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.074425    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-web-config\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.074517 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.074496    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16dab515-6397-4259-bfff-7cb9e0bd83de-config-out\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.074573 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.074540    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.074573 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.074560    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16dab515-6397-4259-bfff-7cb9e0bd83de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.074673 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.074584    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.175823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.175779    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.175914 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.175835    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/16dab515-6397-4259-bfff-7cb9e0bd83de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.175914 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.175858    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-config-volume\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.175914 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.175875    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.176025 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.175917    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16dab515-6397-4259-bfff-7cb9e0bd83de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.176025 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.175965    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhhzx\" (UniqueName: \"kubernetes.io/projected/16dab515-6397-4259-bfff-7cb9e0bd83de-kube-api-access-rhhzx\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.176025 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.175993    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.176170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.176043    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16dab515-6397-4259-bfff-7cb9e0bd83de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.176170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.176066    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-web-config\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.176170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.176089    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16dab515-6397-4259-bfff-7cb9e0bd83de-config-out\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.176170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.176130    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.176170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.176157    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16dab515-6397-4259-bfff-7cb9e0bd83de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.176437 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.176182    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.176437 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.176271    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/16dab515-6397-4259-bfff-7cb9e0bd83de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.178123 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.177145    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16dab515-6397-4259-bfff-7cb9e0bd83de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.178123 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.177760    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16dab515-6397-4259-bfff-7cb9e0bd83de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.179170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.178961    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.179170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.179021    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16dab515-6397-4259-bfff-7cb9e0bd83de-config-out\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.179170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.179068    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-config-volume\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.179170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.179077    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.179170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.179118    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.179170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.179136    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.179411 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.179379    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.180159 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.180138    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16dab515-6397-4259-bfff-7cb9e0bd83de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.180962 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.180941    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16dab515-6397-4259-bfff-7cb9e0bd83de-web-config\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.229623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.229599    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhhzx\" (UniqueName: \"kubernetes.io/projected/16dab515-6397-4259-bfff-7cb9e0bd83de-kube-api-access-rhhzx\") pod \"alertmanager-main-0\" (UID: \"16dab515-6397-4259-bfff-7cb9e0bd83de\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.232437 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.232421    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 03:08:30.693290 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:30.693262    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 03:08:30.695547 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:08:30.695522    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16dab515_6397_4259_bfff_7cb9e0bd83de.slice/crio-4d037ce31a267d979391a80cde2a770988c1834fcb8a9651550d3ac9beb74f1e WatchSource:0}: Error finding container 4d037ce31a267d979391a80cde2a770988c1834fcb8a9651550d3ac9beb74f1e: Status 404 returned error can't find the container with id 4d037ce31a267d979391a80cde2a770988c1834fcb8a9651550d3ac9beb74f1e
Oct 31 03:08:31.150012 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:31.149980    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13cce5e-6c03-4d1b-9295-0a15df7403d5" path="/var/lib/kubelet/pods/a13cce5e-6c03-4d1b-9295-0a15df7403d5/volumes"
Oct 31 03:08:31.456581 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:31.456493    4900 generic.go:356] "Generic (PLEG): container finished" podID="16dab515-6397-4259-bfff-7cb9e0bd83de" containerID="873a22e2288ecfd449339c2cd20209dc7eb135508ae8802c5b1c4b6040170d53" exitCode=0
Oct 31 03:08:31.456581 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:31.456535    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16dab515-6397-4259-bfff-7cb9e0bd83de","Type":"ContainerDied","Data":"873a22e2288ecfd449339c2cd20209dc7eb135508ae8802c5b1c4b6040170d53"}
Oct 31 03:08:31.456769 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:31.456601    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16dab515-6397-4259-bfff-7cb9e0bd83de","Type":"ContainerStarted","Data":"4d037ce31a267d979391a80cde2a770988c1834fcb8a9651550d3ac9beb74f1e"}
Oct 31 03:08:32.463140 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:32.463104    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16dab515-6397-4259-bfff-7cb9e0bd83de","Type":"ContainerStarted","Data":"1b05802fa1b76bdcf685043b13482aa7cba2efc4435130f688deba09ee4fefdb"}
Oct 31 03:08:32.463612 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:32.463178    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16dab515-6397-4259-bfff-7cb9e0bd83de","Type":"ContainerStarted","Data":"edb3767ac0aceba75e8aaec8475cd141a8884a2ae7f7d1ef77f19b0f3ec983b0"}
Oct 31 03:08:32.463612 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:32.463223    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16dab515-6397-4259-bfff-7cb9e0bd83de","Type":"ContainerStarted","Data":"8cf2a8e084d5f60497993436acc9d7c04600beaa285bf829eba029d9a3b707d1"}
Oct 31 03:08:32.463612 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:32.463264    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16dab515-6397-4259-bfff-7cb9e0bd83de","Type":"ContainerStarted","Data":"de4ee9e69e08da456bcdcd0552a238daf93db7d4fd855dec6dc7ae7a3a47bdf6"}
Oct 31 03:08:32.463612 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:32.463311    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16dab515-6397-4259-bfff-7cb9e0bd83de","Type":"ContainerStarted","Data":"26a1f4e11d8c527b7175c6137887549ebb8e0a9007c4c5860deaad0c9606ece4"}
Oct 31 03:08:32.463612 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:32.463361    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"16dab515-6397-4259-bfff-7cb9e0bd83de","Type":"ContainerStarted","Data":"bacc4a165503629e540b5864f72ac48e70cce20c561dc370f31db77111eca989"}
Oct 31 03:08:32.581928 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:32.581874    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.570231171 podStartE2EDuration="3.581860555s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.629384ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:08:32.58129768 +0000 UTC m=+65.547631051" watchObservedRunningTime="2026-10-31 03:08:32.581860555 +0000 UTC m=+65.548193898"
Oct 31 03:08:32.589647 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:32.589625    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:32.589725 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:32.589684    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:32.594072 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:32.594049    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:33.147155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:33.147115    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:33.354057 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:33.354018    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:33.471135 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:33.471067    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b4dcb7b66-t7rrk"
Oct 31 03:08:33.628327 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:33.628298    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f74698894-dkrft"]
Oct 31 03:08:36.982759 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:08:36.982726    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:08:37.438775 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:37.438699    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:37.643469 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:37.643436    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:41.993532 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:41.993496    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:50.049402 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:50.049369    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:08:58.647864 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:58.647827    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7f74698894-dkrft" podUID="d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1" containerName="console" containerID="cri-o://d0833e2ebd8b8d0a61654a3c8c8bdf62d95a84438aec6b5a43d3488ee5069c07" gracePeriod=15
Oct 31 03:08:58.893633 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:58.893608    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:08:59.004155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.004129    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-config\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-config\") pod \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") "
Oct 31 03:08:59.004332 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.004171    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-trusted-ca-bundle\") pod \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") "
Oct 31 03:08:59.004332 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.004209    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-serving-cert\") pod \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") "
Oct 31 03:08:59.004453 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.004354    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-oauth-config\") pod \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") "
Oct 31 03:08:59.004453 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.004406    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-oauth-serving-cert\") pod \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") "
Oct 31 03:08:59.004897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.004462    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-tmp\") pod \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") "
Oct 31 03:08:59.004897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.004493    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-kube-api-access-ntz4r\" (UniqueName: \"kubernetes.io/projected/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-kube-api-access-ntz4r\") pod \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") "
Oct 31 03:08:59.005231 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.004592    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-config" pod "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1" (UID: "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:59.005231 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.004649    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-trusted-ca-bundle" pod "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1" (UID: "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:59.005231 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.005206    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-service-ca\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-service-ca\") pod \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\" (UID: \"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1\") "
Oct 31 03:08:59.005565 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.005094    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-tmp" pod "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1" (UID: "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:08:59.005565 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.005102    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-oauth-serving-cert" pod "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1" (UID: "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:59.005565 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.005514    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-service-ca" pod "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1" (UID: "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:08:59.005916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.005813    4900 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-tmp\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:59.005916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.005844    4900 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:59.005916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.005861    4900 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-config\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:59.005916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.005877    4900 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-trusted-ca-bundle\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:59.005916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.005893    4900 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-oauth-serving-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:59.007526 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.007498    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-serving-cert" pod "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1" (UID: "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:59.007602 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.007552    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-oauth-config" pod "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1" (UID: "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:08:59.007710 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.007694    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-kube-api-access-ntz4r" pod "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1" (UID: "d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1"). InnerVolumeSpecName "kube-api-access-ntz4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:08:59.107151 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.107116    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntz4r\" (UniqueName: \"kubernetes.io/projected/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-kube-api-access-ntz4r\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:59.107151 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.107150    4900 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-serving-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:59.107151 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.107160    4900 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1-console-oauth-config\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:08:59.561417 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.561383    4900 generic.go:356] "Generic (PLEG): container finished" podID="d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1" containerID="d0833e2ebd8b8d0a61654a3c8c8bdf62d95a84438aec6b5a43d3488ee5069c07" exitCode=0
Oct 31 03:08:59.561586 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.561431    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f74698894-dkrft"
Oct 31 03:08:59.561586 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.561476    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f74698894-dkrft" event={"ID":"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1","Type":"ContainerDied","Data":"d0833e2ebd8b8d0a61654a3c8c8bdf62d95a84438aec6b5a43d3488ee5069c07"}
Oct 31 03:08:59.561586 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.561529    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f74698894-dkrft" event={"ID":"d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1","Type":"ContainerDied","Data":"dc6b916928d2c34c1b0382a6052caf80e565a9e9b557ab3168b855cbe7b7f14f"}
Oct 31 03:08:59.561586 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.561551    4900 scope.go:122] "RemoveContainer" containerID="d0833e2ebd8b8d0a61654a3c8c8bdf62d95a84438aec6b5a43d3488ee5069c07"
Oct 31 03:08:59.569195 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.569175    4900 scope.go:122] "RemoveContainer" containerID="d0833e2ebd8b8d0a61654a3c8c8bdf62d95a84438aec6b5a43d3488ee5069c07"
Oct 31 03:08:59.569453 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:08:59.569437    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0833e2ebd8b8d0a61654a3c8c8bdf62d95a84438aec6b5a43d3488ee5069c07\": container with ID starting with d0833e2ebd8b8d0a61654a3c8c8bdf62d95a84438aec6b5a43d3488ee5069c07 not found: ID does not exist" containerID="d0833e2ebd8b8d0a61654a3c8c8bdf62d95a84438aec6b5a43d3488ee5069c07"
Oct 31 03:08:59.569495 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.569460    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0833e2ebd8b8d0a61654a3c8c8bdf62d95a84438aec6b5a43d3488ee5069c07"} err="failed to get container status \"d0833e2ebd8b8d0a61654a3c8c8bdf62d95a84438aec6b5a43d3488ee5069c07\": rpc error: code = NotFound desc = could not find container \"d0833e2ebd8b8d0a61654a3c8c8bdf62d95a84438aec6b5a43d3488ee5069c07\": container with ID starting with d0833e2ebd8b8d0a61654a3c8c8bdf62d95a84438aec6b5a43d3488ee5069c07 not found: ID does not exist"
Oct 31 03:08:59.712942 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.712907    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f74698894-dkrft"]
Oct 31 03:08:59.759143 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:08:59.759119    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f74698894-dkrft"]
Oct 31 03:09:01.149530 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:09:01.149494    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1" path="/var/lib/kubelet/pods/d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1/volumes"
Oct 31 03:09:04.805535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:09:04.805498    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:09:05.008200 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:09:05.008168    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:09:06.983050 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:09:06.983010    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:09:33.804360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:09:33.804303    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:09:34.037297 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:09:34.037259    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:09:36.982521 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:09:36.982480    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:09:42.320089 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:09:42.320026    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:01.063835 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:01.063782    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:06.982626 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:10:06.982589    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:10:08.213337 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:08.213287    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:08.667480 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:08.667448    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:10.756386 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.756349    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"]
Oct 31 03:10:10.756908 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.756807    4900 cpu_manager.go:414] "containerMap: removing container" podUID="d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1" containerName="console"
Oct 31 03:10:10.756908 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.756824    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1" containerName="console"
Oct 31 03:10:10.757044 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.756923    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="d722c6fb-13d0-47fe-aee9-8b1fb73fa3d1" containerName="console"
Oct 31 03:10:10.759753 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.759733    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.818125 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.818090    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 03:10:10.818125 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.818117    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 03:10:10.818333 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.818202    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 03:10:10.852718 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.852685    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.852900 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.852744    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.852900 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.852869    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.853070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.852908    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.853070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.852938    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.853070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.853009    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.853169 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.853092    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cbbq\" (UniqueName: \"kubernetes.io/projected/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-kube-api-access-2cbbq\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.860371 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.860343    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 03:10:10.860429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.860343    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 03:10:10.860583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.860568    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 03:10:10.860618 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.860587    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-r7bff\""
Oct 31 03:10:10.868154 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.868128    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"]
Oct 31 03:10:10.929189 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.929157    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:10.954532 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.954500    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.954721 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.954548    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.954721 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.954586    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.954721 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.954618    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.954721 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.954647    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.954721 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.954677    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.955028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.954767    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cbbq\" (UniqueName: \"kubernetes.io/projected/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-kube-api-access-2cbbq\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.955320 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.955297    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.957028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.957001    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.957300 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.957272    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.957400 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.957387    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:10.957472 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:10.957453    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:11.015473 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:11.015396    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:11.015602 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:11.015477    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cbbq\" (UniqueName: \"kubernetes.io/projected/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-kube-api-access-2cbbq\") pod \"istiod-openshift-gateway-7876bfcb74-2kz8v\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:11.071215 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:11.071179    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:11.223308 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:11.223244    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:11.493874 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:11.493717    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"]
Oct 31 03:10:11.496413 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:10:11.496388    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb8f1a8_0443_4f97_8ba6_d0c6a63c5b67.slice/crio-553b0f9a955233a22a68aa34102320660a4cc4eaea938486f479478539176258 WatchSource:0}: Error finding container 553b0f9a955233a22a68aa34102320660a4cc4eaea938486f479478539176258: Status 404 returned error can't find the container with id 553b0f9a955233a22a68aa34102320660a4cc4eaea938486f479478539176258
Oct 31 03:10:11.808828 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:11.808735    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v" event={"ID":"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67","Type":"ContainerStarted","Data":"553b0f9a955233a22a68aa34102320660a4cc4eaea938486f479478539176258"}
Oct 31 03:10:12.459882 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:12.459838    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:15.517739 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:15.517703    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:10:15.518148 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:15.517780    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:10:15.825562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:15.825459    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v" event={"ID":"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67","Type":"ContainerStarted","Data":"bf102e687812f4579e8a62fe99039f0312ab00df056302e7dacfb283419d9f7c"}
Oct 31 03:10:15.825722 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:15.825572    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:16.029099 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:16.029046    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v" podStartSLOduration=2.010290209 podStartE2EDuration="6.029031376s" totalImagesPullingTime="4.018741167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:10:15.982521937 +0000 UTC m=+168.948855282" watchObservedRunningTime="2026-10-31 03:10:16.029031376 +0000 UTC m=+168.995364719"
Oct 31 03:10:16.831029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:16.830989    4900 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-2kz8v container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:10:16.831407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:16.831059    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v" podUID="4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:10:17.015281 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:17.015229    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:17.237216 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:17.237179    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:17.668391 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:17.668281    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:17.890324 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:17.890284    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:18.132037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:18.131997    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:19.830939 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:19.830900    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:10:21.838447 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:21.838415    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:36.982581 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:10:36.982546    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:10:42.695859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:42.695745    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:50.168366 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:50.168324    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:50.398332 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:50.398296    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:58.382796 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:58.382752    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:58.780382 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:58.780285    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:59.106378 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.106291    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:59.265258 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.265220    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"]
Oct 31 03:10:59.275558 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.275529    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.311004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.310971    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"]
Oct 31 03:10:59.341408 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.341362    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/73bc3fc7-c81c-407b-9655-77129757277d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.341523 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.341484    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73bc3fc7-c81c-407b-9655-77129757277d-tmp\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.341608 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.341591    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/73bc3fc7-c81c-407b-9655-77129757277d-manifests\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.341650 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.341623    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73bc3fc7-c81c-407b-9655-77129757277d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.341693 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.341678    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4rx2\" (UniqueName: \"kubernetes.io/projected/73bc3fc7-c81c-407b-9655-77129757277d-kube-api-access-r4rx2\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.341736 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.341727    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/73bc3fc7-c81c-407b-9655-77129757277d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.388084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.388011    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:59.404916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.404893    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 03:10:59.405041 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.404934    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 03:10:59.405138 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.405123    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 03:10:59.405187 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.405155    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 03:10:59.405232 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.405187    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-92w4t\""
Oct 31 03:10:59.405275 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.405243    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 03:10:59.442777 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.442748    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73bc3fc7-c81c-407b-9655-77129757277d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.442942 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.442810    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4rx2\" (UniqueName: \"kubernetes.io/projected/73bc3fc7-c81c-407b-9655-77129757277d-kube-api-access-r4rx2\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.442942 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.442844    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/73bc3fc7-c81c-407b-9655-77129757277d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.442942 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.442875    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/73bc3fc7-c81c-407b-9655-77129757277d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.442942 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.442921    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73bc3fc7-c81c-407b-9655-77129757277d-tmp\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.443171 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.442962    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/73bc3fc7-c81c-407b-9655-77129757277d-manifests\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.443362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.443339    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/73bc3fc7-c81c-407b-9655-77129757277d-manifests\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.445292 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.445264    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/73bc3fc7-c81c-407b-9655-77129757277d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.445383 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.445315    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/73bc3fc7-c81c-407b-9655-77129757277d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.445542 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.445520    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/73bc3fc7-c81c-407b-9655-77129757277d-tmp\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.445737 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.445721    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/73bc3fc7-c81c-407b-9655-77129757277d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.497030 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.496999    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4rx2\" (UniqueName: \"kubernetes.io/projected/73bc3fc7-c81c-407b-9655-77129757277d-kube-api-access-r4rx2\") pod \"opendatahub-operator-controller-manager-6f949c7964-tmk7x\" (UID: \"73bc3fc7-c81c-407b-9655-77129757277d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.590165 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.590128    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:10:59.678605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.678575    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:10:59.940526 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:10:59.940449    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:00.063600 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:00.063569    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"]
Oct 31 03:11:00.066963 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:11:00.066935    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bc3fc7_c81c_407b_9655_77129757277d.slice/crio-fd948a7d33da98f12de7ca5b765e22cff50d9db9e421c42c4e507d4e4976c5d2 WatchSource:0}: Error finding container fd948a7d33da98f12de7ca5b765e22cff50d9db9e421c42c4e507d4e4976c5d2: Status 404 returned error can't find the container with id fd948a7d33da98f12de7ca5b765e22cff50d9db9e421c42c4e507d4e4976c5d2
Oct 31 03:11:00.188806 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:00.188754    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:00.977421 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:00.977388    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x" event={"ID":"73bc3fc7-c81c-407b-9655-77129757277d","Type":"ContainerStarted","Data":"fd948a7d33da98f12de7ca5b765e22cff50d9db9e421c42c4e507d4e4976c5d2"}
Oct 31 03:11:02.178180 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:02.178147    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:02.411335 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:02.411302    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:06.178602 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:06.178572    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:06.983554 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:11:06.983518    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:11:07.432184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:07.432107    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:08.773766 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:08.773732    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:10.923965 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:10.923934    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:11.015143 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:11.015113    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x" event={"ID":"73bc3fc7-c81c-407b-9655-77129757277d","Type":"ContainerStarted","Data":"1f6818988880d8093a635b4a028710b6c2efb84a1d0cf89386baab239f047e66"}
Oct 31 03:11:11.254415 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:11.254386    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:12.019841 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:12.019778    4900 generic.go:356] "Generic (PLEG): container finished" podID="73bc3fc7-c81c-407b-9655-77129757277d" containerID="1f6818988880d8093a635b4a028710b6c2efb84a1d0cf89386baab239f047e66" exitCode=0
Oct 31 03:11:12.019841 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:12.019822    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x" event={"ID":"73bc3fc7-c81c-407b-9655-77129757277d","Type":"ContainerDied","Data":"1f6818988880d8093a635b4a028710b6c2efb84a1d0cf89386baab239f047e66"}
Oct 31 03:11:13.025353 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:13.025309    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x" event={"ID":"73bc3fc7-c81c-407b-9655-77129757277d","Type":"ContainerStarted","Data":"76db9c0ac8df405515e1b207f976d5a3888247f6ec2ac80dfe4adcba05414177"}
Oct 31 03:11:13.025743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:13.025440    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:11:13.242464 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:13.242419    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x" podStartSLOduration=2.125958067 podStartE2EDuration="14.242407323s" totalImagesPullingTime="11.635035227s" totalInitContainerRuntime="481.414029ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:10:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:13.196142288 +0000 UTC m=+226.162475631" watchObservedRunningTime="2026-10-31 03:11:13.242407323 +0000 UTC m=+226.208740668"
Oct 31 03:11:13.619852 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:13.619725    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:14.683335 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:14.683299    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:14.951655 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:14.951579    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:16.181118 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:16.181083    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:16.866384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:16.866352    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:17.367448 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:17.367410    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:17.600635 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:17.600597    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:18.604142 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:18.604111    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:24.032118 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:24.032088    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6f949c7964-tmk7x"
Oct 31 03:11:24.855188 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:24.855147    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:25.580799 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:25.580745    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:25.827212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:25.827174    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:28.757806 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:28.757754    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:29.004622 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:29.004581    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:30.590261 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:30.590225    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:30.834342 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:30.834311    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:31.197227 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:31.197197    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:31.818937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:31.818901    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:32.398569 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:32.398535    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:33.461512 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:33.461479    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:33.706107 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:33.706072    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:34.766515 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:34.766482    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:36.655210 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:36.655173    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:36.879783 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:36.879748    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:36.983257 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:11:36.983223    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:11:37.100105 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:37.100070    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:38.024178 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:38.024142    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:38.263448 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:38.263413    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:41.967482 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:41.967448    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:42.199047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:42.199018    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:42.474307 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:42.474277    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:43.051879 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:43.051848    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:43.867314 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:43.867276    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:44.055328 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.055289    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j"]
Oct 31 03:11:44.058435 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.058415    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j"
Oct 31 03:11:44.109725 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.109697    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-csxjl\""
Oct 31 03:11:44.154735 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.154670    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:44.157418 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.157398    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j"]
Oct 31 03:11:44.169549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.169526    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 03:11:44.169549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.169541    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 03:11:44.213819 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.213771    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwrmz\" (UniqueName: \"kubernetes.io/projected/b158a0dc-55bd-4d81-bf1c-f83d4896dba5-kube-api-access-qwrmz\") pod \"kube-auth-proxy-cb6b5fd89-vj58j\" (UID: \"b158a0dc-55bd-4d81-bf1c-f83d4896dba5\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j"
Oct 31 03:11:44.213940 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.213845    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b158a0dc-55bd-4d81-bf1c-f83d4896dba5-tmp\") pod \"kube-auth-proxy-cb6b5fd89-vj58j\" (UID: \"b158a0dc-55bd-4d81-bf1c-f83d4896dba5\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j"
Oct 31 03:11:44.214002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.213965    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b158a0dc-55bd-4d81-bf1c-f83d4896dba5-tls-certs\") pod \"kube-auth-proxy-cb6b5fd89-vj58j\" (UID: \"b158a0dc-55bd-4d81-bf1c-f83d4896dba5\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j"
Oct 31 03:11:44.314645 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.314612    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwrmz\" (UniqueName: \"kubernetes.io/projected/b158a0dc-55bd-4d81-bf1c-f83d4896dba5-kube-api-access-qwrmz\") pod \"kube-auth-proxy-cb6b5fd89-vj58j\" (UID: \"b158a0dc-55bd-4d81-bf1c-f83d4896dba5\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j"
Oct 31 03:11:44.314823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.314674    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b158a0dc-55bd-4d81-bf1c-f83d4896dba5-tmp\") pod \"kube-auth-proxy-cb6b5fd89-vj58j\" (UID: \"b158a0dc-55bd-4d81-bf1c-f83d4896dba5\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j"
Oct 31 03:11:44.314823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.314725    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b158a0dc-55bd-4d81-bf1c-f83d4896dba5-tls-certs\") pod \"kube-auth-proxy-cb6b5fd89-vj58j\" (UID: \"b158a0dc-55bd-4d81-bf1c-f83d4896dba5\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j"
Oct 31 03:11:44.317336 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.317308    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b158a0dc-55bd-4d81-bf1c-f83d4896dba5-tls-certs\") pod \"kube-auth-proxy-cb6b5fd89-vj58j\" (UID: \"b158a0dc-55bd-4d81-bf1c-f83d4896dba5\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j"
Oct 31 03:11:44.317466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.317339    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b158a0dc-55bd-4d81-bf1c-f83d4896dba5-tmp\") pod \"kube-auth-proxy-cb6b5fd89-vj58j\" (UID: \"b158a0dc-55bd-4d81-bf1c-f83d4896dba5\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j"
Oct 31 03:11:44.377491 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.377457    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwrmz\" (UniqueName: \"kubernetes.io/projected/b158a0dc-55bd-4d81-bf1c-f83d4896dba5-kube-api-access-qwrmz\") pod \"kube-auth-proxy-cb6b5fd89-vj58j\" (UID: \"b158a0dc-55bd-4d81-bf1c-f83d4896dba5\") " pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j"
Oct 31 03:11:44.436853 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.436757    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:44.672056 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.672023    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j"
Oct 31 03:11:44.712452 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.712366    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:44.953084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:44.953019    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:45.040465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:45.040432    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j"]
Oct 31 03:11:45.043900 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:11:45.043872    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb158a0dc_55bd_4d81_bf1c_f83d4896dba5.slice/crio-575015a91bdf9dce48e165b99108448847f60dbeb9456f7797cd607e29eb4ecc WatchSource:0}: Error finding container 575015a91bdf9dce48e165b99108448847f60dbeb9456f7797cd607e29eb4ecc: Status 404 returned error can't find the container with id 575015a91bdf9dce48e165b99108448847f60dbeb9456f7797cd607e29eb4ecc
Oct 31 03:11:45.137193 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:45.137154    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j" event={"ID":"b158a0dc-55bd-4d81-bf1c-f83d4896dba5","Type":"ContainerStarted","Data":"575015a91bdf9dce48e165b99108448847f60dbeb9456f7797cd607e29eb4ecc"}
Oct 31 03:11:45.242368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:45.242337    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:46.412880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:46.412843    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:47.483182 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.483139    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"]
Oct 31 03:11:47.491830 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.491807    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:47.591654 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.591625    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-w6dx4\""
Oct 31 03:11:47.591997 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.591945    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 03:11:47.591997 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.591972    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 31 03:11:47.591997 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.591989    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 31 03:11:47.591997 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.591960    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:11:47.592298 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.592038    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 31 03:11:47.593211 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.593192    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"]
Oct 31 03:11:47.644849 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.644814    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b68a89-a877-4b85-9d90-b8c48ce3e639-metrics-certs\") pod \"jobset-controller-manager-6db6df8475-ktgnj\" (UID: \"86b68a89-a877-4b85-9d90-b8c48ce3e639\") " pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:47.644994 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.644880    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86b68a89-a877-4b85-9d90-b8c48ce3e639-cert\") pod \"jobset-controller-manager-6db6df8475-ktgnj\" (UID: \"86b68a89-a877-4b85-9d90-b8c48ce3e639\") " pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:47.645082 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.644993    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/86b68a89-a877-4b85-9d90-b8c48ce3e639-manager-config\") pod \"jobset-controller-manager-6db6df8475-ktgnj\" (UID: \"86b68a89-a877-4b85-9d90-b8c48ce3e639\") " pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:47.645082 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.645023    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmn7g\" (UniqueName: \"kubernetes.io/projected/86b68a89-a877-4b85-9d90-b8c48ce3e639-kube-api-access-mmn7g\") pod \"jobset-controller-manager-6db6df8475-ktgnj\" (UID: \"86b68a89-a877-4b85-9d90-b8c48ce3e639\") " pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:47.745973 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.745886    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/86b68a89-a877-4b85-9d90-b8c48ce3e639-manager-config\") pod \"jobset-controller-manager-6db6df8475-ktgnj\" (UID: \"86b68a89-a877-4b85-9d90-b8c48ce3e639\") " pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:47.745973 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.745929    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmn7g\" (UniqueName: \"kubernetes.io/projected/86b68a89-a877-4b85-9d90-b8c48ce3e639-kube-api-access-mmn7g\") pod \"jobset-controller-manager-6db6df8475-ktgnj\" (UID: \"86b68a89-a877-4b85-9d90-b8c48ce3e639\") " pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:47.746188 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.746001    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b68a89-a877-4b85-9d90-b8c48ce3e639-metrics-certs\") pod \"jobset-controller-manager-6db6df8475-ktgnj\" (UID: \"86b68a89-a877-4b85-9d90-b8c48ce3e639\") " pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:47.746188 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.746042    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86b68a89-a877-4b85-9d90-b8c48ce3e639-cert\") pod \"jobset-controller-manager-6db6df8475-ktgnj\" (UID: \"86b68a89-a877-4b85-9d90-b8c48ce3e639\") " pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:47.746751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.746725    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/86b68a89-a877-4b85-9d90-b8c48ce3e639-manager-config\") pod \"jobset-controller-manager-6db6df8475-ktgnj\" (UID: \"86b68a89-a877-4b85-9d90-b8c48ce3e639\") " pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:47.748652 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.748628    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86b68a89-a877-4b85-9d90-b8c48ce3e639-metrics-certs\") pod \"jobset-controller-manager-6db6df8475-ktgnj\" (UID: \"86b68a89-a877-4b85-9d90-b8c48ce3e639\") " pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:47.748909 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.748886    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86b68a89-a877-4b85-9d90-b8c48ce3e639-cert\") pod \"jobset-controller-manager-6db6df8475-ktgnj\" (UID: \"86b68a89-a877-4b85-9d90-b8c48ce3e639\") " pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:47.801057 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.801025    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmn7g\" (UniqueName: \"kubernetes.io/projected/86b68a89-a877-4b85-9d90-b8c48ce3e639-kube-api-access-mmn7g\") pod \"jobset-controller-manager-6db6df8475-ktgnj\" (UID: \"86b68a89-a877-4b85-9d90-b8c48ce3e639\") " pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:47.805781 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:47.805747    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:48.150714 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:48.150635    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j" event={"ID":"b158a0dc-55bd-4d81-bf1c-f83d4896dba5","Type":"ContainerStarted","Data":"49d0204d3a8d00b8d7c334e65f591f6d50bd213a292a22099c7c81a24a1dfad4"}
Oct 31 03:11:48.185530 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:48.185503    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"]
Oct 31 03:11:48.188221 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:11:48.188195    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b68a89_a877_4b85_9d90_b8c48ce3e639.slice/crio-82f877345ee12a165e643a2cb0d33a826550a0b981f8f35468a0dfee88e8753b WatchSource:0}: Error finding container 82f877345ee12a165e643a2cb0d33a826550a0b981f8f35468a0dfee88e8753b: Status 404 returned error can't find the container with id 82f877345ee12a165e643a2cb0d33a826550a0b981f8f35468a0dfee88e8753b
Oct 31 03:11:48.258848 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:48.258771    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cb6b5fd89-vj58j" podStartSLOduration=1.35720462 podStartE2EDuration="4.258754561s" totalImagesPullingTime="2.901549941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:48.258253594 +0000 UTC m=+261.224586939" watchObservedRunningTime="2026-10-31 03:11:48.258754561 +0000 UTC m=+261.225087904"
Oct 31 03:11:48.390749 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:48.390711    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:48.617517 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:48.617469    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:49.154693 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:49.154658    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj" event={"ID":"86b68a89-a877-4b85-9d90-b8c48ce3e639","Type":"ContainerStarted","Data":"82f877345ee12a165e643a2cb0d33a826550a0b981f8f35468a0dfee88e8753b"}
Oct 31 03:11:51.778245 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:51.778206    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:53.170746 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:53.170662    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj" event={"ID":"86b68a89-a877-4b85-9d90-b8c48ce3e639","Type":"ContainerStarted","Data":"8cdc7507fd248ebee1ac6b358fb73227792a9395ab5f8ef85c49063ed45f401c"}
Oct 31 03:11:53.171131 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:53.170821    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:11:53.191998 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:53.191967    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:53.279476 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:53.279412    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj" podStartSLOduration=1.7320435280000002 podStartE2EDuration="6.279389995s" totalImagesPullingTime="4.547346467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:11:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:11:53.27899545 +0000 UTC m=+266.245328796" watchObservedRunningTime="2026-10-31 03:11:53.279389995 +0000 UTC m=+266.245723340"
Oct 31 03:11:53.403889 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:53.403857    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:58.039037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:58.038999    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:11:58.289368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:11:58.289292    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:04.179930 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:04.179899    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6db6df8475-ktgnj"
Oct 31 03:12:04.523623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:04.523594    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:04.727394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:04.727359    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:06.982192 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:12:06.982147    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:12:09.288490 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:09.288410    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:09.503128 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:09.503094    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:14.071013 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:14.070980    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:14.285377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:14.285339    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:18.998348 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:18.998309    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:19.360448 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:19.360361    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:19.585514 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:19.585467    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:19.795990 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:19.795958    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:20.000428 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:20.000392    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:20.371825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:20.371771    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:20.584767 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:20.584724    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:23.962313 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:23.962268    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:24.177562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:24.177529    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:26.336328 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:26.336295    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:26.537761 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:26.537726    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:27.151513 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:27.151488    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:12:27.151684 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:27.151495    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:12:31.135043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:31.135010    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:31.345522 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:31.345482    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:35.885003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:35.884965    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:36.110441 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:36.110408    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:36.983371 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:12:36.983335    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:12:40.390073 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:40.390038    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:40.617575 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:40.617537    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:41.191354 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:41.191321    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:12:43.347322 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:12:43.347291    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:02.847901 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:02.847855    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:03.277490 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:03.277457    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:03.494918 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:03.494887    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:06.982426 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:13:06.982394    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:13:12.107141 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:12.107108    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:12.334534 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:12.334500    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:15.908424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:15.908390    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:16.130805 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:16.130759    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:16.340076 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:16.340041    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:16.584901 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:16.584861    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:16.798366 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:16.798334    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:17.012530 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:17.012490    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:18.034706 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:18.034665    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:22.454700 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.454663    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"]
Oct 31 03:13:22.455132 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.454924    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v" podUID="4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67" containerName="discovery" containerID="cri-o://bf102e687812f4579e8a62fe99039f0312ab00df056302e7dacfb283419d9f7c" gracePeriod=30
Oct 31 03:13:22.692228 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.692200    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:13:22.774062 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.774028    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-kubeconfig\") pod \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") "
Oct 31 03:13:22.774062 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.774066    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-cacerts\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-cacerts\") pod \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") "
Oct 31 03:13:22.774317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.774112    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-local-certs\") pod \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") "
Oct 31 03:13:22.774317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.774134    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-csr-dns-cert\") pod \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") "
Oct 31 03:13:22.774317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.774186    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-token\" (UniqueName: \"kubernetes.io/projected/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-token\") pod \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") "
Oct 31 03:13:22.774317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.774225    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-kube-api-access-2cbbq\" (UniqueName: \"kubernetes.io/projected/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-kube-api-access-2cbbq\") pod \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") "
Oct 31 03:13:22.774317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.774261    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-csr-ca-configmap\") pod \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\" (UID: \"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67\") "
Oct 31 03:13:22.774944 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.774905    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-csr-ca-configmap" pod "4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67" (UID: "4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:13:22.776560 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.776529    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-kubeconfig" pod "4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67" (UID: "4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:13:22.776669 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.776587    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-cacerts" pod "4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67" (UID: "4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:13:22.776737 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.776686    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-token" pod "4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67" (UID: "4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:13:22.776820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.776778    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-local-certs" pod "4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67" (UID: "4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:13:22.776820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.776808    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-kube-api-access-2cbbq" pod "4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67" (UID: "4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67"). InnerVolumeSpecName "kube-api-access-2cbbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:13:22.776897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.776836    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-csr-dns-cert" pod "4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67" (UID: "4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:13:22.819824 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.819772    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:22.875226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.875188    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:13:22.875226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.875221    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cbbq\" (UniqueName: \"kubernetes.io/projected/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-kube-api-access-2cbbq\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:13:22.875431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.875236    4900 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-csr-ca-configmap\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:13:22.875431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.875248    4900 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-kubeconfig\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:13:22.875431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.875261    4900 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-cacerts\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:13:22.875431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.875273    4900 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-local-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:13:22.875431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:22.875285    4900 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67-istio-csr-dns-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:13:23.016672 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:23.016636    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:23.254436 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:23.254380    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:23.480548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:23.480452    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:23.486267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:23.486239    4900 generic.go:356] "Generic (PLEG): container finished" podID="4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67" containerID="bf102e687812f4579e8a62fe99039f0312ab00df056302e7dacfb283419d9f7c" exitCode=0
Oct 31 03:13:23.486401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:23.486283    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"
Oct 31 03:13:23.486401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:23.486327    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v" event={"ID":"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67","Type":"ContainerDied","Data":"bf102e687812f4579e8a62fe99039f0312ab00df056302e7dacfb283419d9f7c"}
Oct 31 03:13:23.486401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:23.486378    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v" event={"ID":"4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67","Type":"ContainerDied","Data":"553b0f9a955233a22a68aa34102320660a4cc4eaea938486f479478539176258"}
Oct 31 03:13:23.486558 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:23.486405    4900 scope.go:122] "RemoveContainer" containerID="bf102e687812f4579e8a62fe99039f0312ab00df056302e7dacfb283419d9f7c"
Oct 31 03:13:23.494641 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:23.494621    4900 scope.go:122] "RemoveContainer" containerID="bf102e687812f4579e8a62fe99039f0312ab00df056302e7dacfb283419d9f7c"
Oct 31 03:13:23.494953 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:13:23.494934    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf102e687812f4579e8a62fe99039f0312ab00df056302e7dacfb283419d9f7c\": container with ID starting with bf102e687812f4579e8a62fe99039f0312ab00df056302e7dacfb283419d9f7c not found: ID does not exist" containerID="bf102e687812f4579e8a62fe99039f0312ab00df056302e7dacfb283419d9f7c"
Oct 31 03:13:23.495027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:23.494965    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf102e687812f4579e8a62fe99039f0312ab00df056302e7dacfb283419d9f7c"} err="failed to get container status \"bf102e687812f4579e8a62fe99039f0312ab00df056302e7dacfb283419d9f7c\": rpc error: code = NotFound desc = could not find container \"bf102e687812f4579e8a62fe99039f0312ab00df056302e7dacfb283419d9f7c\": container with ID starting with bf102e687812f4579e8a62fe99039f0312ab00df056302e7dacfb283419d9f7c not found: ID does not exist"
Oct 31 03:13:23.666568 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:23.666533    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"]
Oct 31 03:13:23.673862 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:23.673832    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-2kz8v"]
Oct 31 03:13:23.744758 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:23.744716    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:23.963377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:23.963341    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:25.149805 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:25.149766    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67" path="/var/lib/kubelet/pods/4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67/volumes"
Oct 31 03:13:36.983026 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:13:36.982989    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:13:41.012115 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:41.012035    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:41.241083 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:41.241048    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:43.114215 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:43.114174    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:43.319838 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:43.319778    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:43.566035 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:43.566003    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:43.782371 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:43.782329    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:44.097938 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.097903    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"]
Oct 31 03:13:44.098220 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.098208    4900 cpu_manager.go:414] "containerMap: removing container" podUID="4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67" containerName="discovery"
Oct 31 03:13:44.098274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.098221    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67" containerName="discovery"
Oct 31 03:13:44.098318 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.098289    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb8f1a8-0443-4f97-8ba6-d0c6a63c5b67" containerName="discovery"
Oct 31 03:13:44.101027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.101010    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:44.141473 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.141435    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a876178-dc85-41ea-8087-4533f316e2f4-openshift-service-ca\") pod \"maas-api-586d94f9c6-h8qlw\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") " pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:44.141911 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.141491    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/6a876178-dc85-41ea-8087-4533f316e2f4-kube-api-access-v4qkr\") pod \"maas-api-586d94f9c6-h8qlw\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") " pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:44.141911 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.141579    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a876178-dc85-41ea-8087-4533f316e2f4-metrics-certs\") pod \"maas-api-586d94f9c6-h8qlw\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") " pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:44.141911 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.141620    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a876178-dc85-41ea-8087-4533f316e2f4-maas-api-tls\") pod \"maas-api-586d94f9c6-h8qlw\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") " pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:44.197101 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.197073    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:13:44.197101 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.197091    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:13:44.197311 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.197107    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-jsn62\""
Oct 31 03:13:44.197311 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.197137    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 03:13:44.197311 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.197205    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 03:13:44.198139 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.198119    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"]
Oct 31 03:13:44.242596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.242560    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a876178-dc85-41ea-8087-4533f316e2f4-maas-api-tls\") pod \"maas-api-586d94f9c6-h8qlw\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") " pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:44.242770 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.242619    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a876178-dc85-41ea-8087-4533f316e2f4-openshift-service-ca\") pod \"maas-api-586d94f9c6-h8qlw\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") " pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:44.242770 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.242695    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/6a876178-dc85-41ea-8087-4533f316e2f4-kube-api-access-v4qkr\") pod \"maas-api-586d94f9c6-h8qlw\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") " pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:44.242929 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.242903    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a876178-dc85-41ea-8087-4533f316e2f4-metrics-certs\") pod \"maas-api-586d94f9c6-h8qlw\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") " pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:44.243439 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.243404    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a876178-dc85-41ea-8087-4533f316e2f4-openshift-service-ca\") pod \"maas-api-586d94f9c6-h8qlw\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") " pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:44.250366 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.250341    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a876178-dc85-41ea-8087-4533f316e2f4-maas-api-tls\") pod \"maas-api-586d94f9c6-h8qlw\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") " pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:44.250765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.250742    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a876178-dc85-41ea-8087-4533f316e2f4-metrics-certs\") pod \"maas-api-586d94f9c6-h8qlw\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") " pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:44.295945 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.295917    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/6a876178-dc85-41ea-8087-4533f316e2f4-kube-api-access-v4qkr\") pod \"maas-api-586d94f9c6-h8qlw\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") " pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:44.412912 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.412831    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:44.776365 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.776331    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"]
Oct 31 03:13:44.779804 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:13:44.779758    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a876178_dc85_41ea_8087_4533f316e2f4.slice/crio-2d4630f2e2ea58409aa85e93cec50acbec03170f02707b1efdcea159f358bbfc WatchSource:0}: Error finding container 2d4630f2e2ea58409aa85e93cec50acbec03170f02707b1efdcea159f358bbfc: Status 404 returned error can't find the container with id 2d4630f2e2ea58409aa85e93cec50acbec03170f02707b1efdcea159f358bbfc
Oct 31 03:13:44.781086 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:44.781067    4900 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:13:45.565751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:45.565718    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw" event={"ID":"6a876178-dc85-41ea-8087-4533f316e2f4","Type":"ContainerStarted","Data":"2d4630f2e2ea58409aa85e93cec50acbec03170f02707b1efdcea159f358bbfc"}
Oct 31 03:13:52.594085 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:52.594047    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw" event={"ID":"6a876178-dc85-41ea-8087-4533f316e2f4","Type":"ContainerStarted","Data":"66d5f837a2d49ebd86ce6b24a862cc7ddcd1855e65aacdd1fcb33d58796cae45"}
Oct 31 03:13:52.594474 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:52.594181    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:52.701405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:52.701330    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw" podStartSLOduration=1.66384454 podStartE2EDuration="8.701314414s" totalImagesPullingTime="7.037469874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:13:52.70075813 +0000 UTC m=+385.667091465" watchObservedRunningTime="2026-10-31 03:13:52.701314414 +0000 UTC m=+385.667647759"
Oct 31 03:13:58.603484 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:58.603455    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:13:59.194506 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.194438    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:59.536103 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.536041    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:13:59.593067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.593041    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-997c84955-lk8d7"]
Oct 31 03:13:59.596238 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.596221    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:13:59.650487 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.650456    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-997c84955-lk8d7"]
Oct 31 03:13:59.665926 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.665902    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f14a322-a62a-464a-bbb5-5c9e60320977-maas-api-tls\") pod \"maas-api-997c84955-lk8d7\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") " pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:13:59.666033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.665940    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f14a322-a62a-464a-bbb5-5c9e60320977-metrics-certs\") pod \"maas-api-997c84955-lk8d7\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") " pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:13:59.666033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.665968    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f14a322-a62a-464a-bbb5-5c9e60320977-openshift-service-ca\") pod \"maas-api-997c84955-lk8d7\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") " pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:13:59.666111 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.666066    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svm9q\" (UniqueName: \"kubernetes.io/projected/9f14a322-a62a-464a-bbb5-5c9e60320977-kube-api-access-svm9q\") pod \"maas-api-997c84955-lk8d7\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") " pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:13:59.767123 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.767095    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f14a322-a62a-464a-bbb5-5c9e60320977-maas-api-tls\") pod \"maas-api-997c84955-lk8d7\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") " pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:13:59.767289 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.767132    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f14a322-a62a-464a-bbb5-5c9e60320977-metrics-certs\") pod \"maas-api-997c84955-lk8d7\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") " pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:13:59.767289 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.767159    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f14a322-a62a-464a-bbb5-5c9e60320977-openshift-service-ca\") pod \"maas-api-997c84955-lk8d7\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") " pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:13:59.767289 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.767176    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svm9q\" (UniqueName: \"kubernetes.io/projected/9f14a322-a62a-464a-bbb5-5c9e60320977-kube-api-access-svm9q\") pod \"maas-api-997c84955-lk8d7\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") " pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:13:59.768046 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.768020    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f14a322-a62a-464a-bbb5-5c9e60320977-openshift-service-ca\") pod \"maas-api-997c84955-lk8d7\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") " pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:13:59.769583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.769553    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f14a322-a62a-464a-bbb5-5c9e60320977-metrics-certs\") pod \"maas-api-997c84955-lk8d7\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") " pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:13:59.769656 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.769556    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f14a322-a62a-464a-bbb5-5c9e60320977-maas-api-tls\") pod \"maas-api-997c84955-lk8d7\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") " pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:13:59.819837 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.819770    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svm9q\" (UniqueName: \"kubernetes.io/projected/9f14a322-a62a-464a-bbb5-5c9e60320977-kube-api-access-svm9q\") pod \"maas-api-997c84955-lk8d7\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") " pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:13:59.907904 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:13:59.907877    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:14:00.099275 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:00.099184    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:14:00.271459 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:00.271425    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-997c84955-lk8d7"]
Oct 31 03:14:00.274518 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:14:00.274491    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f14a322_a62a_464a_bbb5_5c9e60320977.slice/crio-9773e6149e953a0ad606e9120f1d9e89388a0f0c24dffdc94623c8f20516f82a WatchSource:0}: Error finding container 9773e6149e953a0ad606e9120f1d9e89388a0f0c24dffdc94623c8f20516f82a: Status 404 returned error can't find the container with id 9773e6149e953a0ad606e9120f1d9e89388a0f0c24dffdc94623c8f20516f82a
Oct 31 03:14:00.624662 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:00.624625    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7" event={"ID":"9f14a322-a62a-464a-bbb5-5c9e60320977","Type":"ContainerStarted","Data":"9773e6149e953a0ad606e9120f1d9e89388a0f0c24dffdc94623c8f20516f82a"}
Oct 31 03:14:01.167756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:01.167728    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:14:01.629922 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:01.629886    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7" event={"ID":"9f14a322-a62a-464a-bbb5-5c9e60320977","Type":"ContainerStarted","Data":"2ecb6e8321237693d2e789b3a109a80635613a862a6d1316a23cb53e43cb6750"}
Oct 31 03:14:01.630071 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:01.629950    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:14:01.741640 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:01.741589    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7" podStartSLOduration=1.915224649 podStartE2EDuration="2.741575422s" totalImagesPullingTime="826.350773ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:13:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:01.741014168 +0000 UTC m=+394.707347509" watchObservedRunningTime="2026-10-31 03:14:01.741575422 +0000 UTC m=+394.707908765"
Oct 31 03:14:06.982437 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:14:06.982402    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:14:07.639400 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:07.639371    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 03:14:07.795775 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:07.795741    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"]
Oct 31 03:14:07.796201 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:07.796013    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw" podUID="6a876178-dc85-41ea-8087-4533f316e2f4" containerName="maas-api" containerID="cri-o://66d5f837a2d49ebd86ce6b24a862cc7ddcd1855e65aacdd1fcb33d58796cae45" gracePeriod=30
Oct 31 03:14:08.039714 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.039690    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:14:08.135492 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.135458    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a876178-dc85-41ea-8087-4533f316e2f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a876178-dc85-41ea-8087-4533f316e2f4-openshift-service-ca\") pod \"6a876178-dc85-41ea-8087-4533f316e2f4\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") "
Oct 31 03:14:08.135684 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.135527    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a876178-dc85-41ea-8087-4533f316e2f4-kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/6a876178-dc85-41ea-8087-4533f316e2f4-kube-api-access-v4qkr\") pod \"6a876178-dc85-41ea-8087-4533f316e2f4\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") "
Oct 31 03:14:08.135684 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.135568    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a876178-dc85-41ea-8087-4533f316e2f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a876178-dc85-41ea-8087-4533f316e2f4-metrics-certs\") pod \"6a876178-dc85-41ea-8087-4533f316e2f4\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") "
Oct 31 03:14:08.135684 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.135605    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a876178-dc85-41ea-8087-4533f316e2f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a876178-dc85-41ea-8087-4533f316e2f4-maas-api-tls\") pod \"6a876178-dc85-41ea-8087-4533f316e2f4\" (UID: \"6a876178-dc85-41ea-8087-4533f316e2f4\") "
Oct 31 03:14:08.135894 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.135875    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a876178-dc85-41ea-8087-4533f316e2f4-openshift-service-ca" pod "6a876178-dc85-41ea-8087-4533f316e2f4" (UID: "6a876178-dc85-41ea-8087-4533f316e2f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:14:08.137745 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.137719    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a876178-dc85-41ea-8087-4533f316e2f4-maas-api-tls" pod "6a876178-dc85-41ea-8087-4533f316e2f4" (UID: "6a876178-dc85-41ea-8087-4533f316e2f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:14:08.137842 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.137758    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a876178-dc85-41ea-8087-4533f316e2f4-kube-api-access-v4qkr" pod "6a876178-dc85-41ea-8087-4533f316e2f4" (UID: "6a876178-dc85-41ea-8087-4533f316e2f4"). InnerVolumeSpecName "kube-api-access-v4qkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:14:08.137917 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.137897    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a876178-dc85-41ea-8087-4533f316e2f4-metrics-certs" pod "6a876178-dc85-41ea-8087-4533f316e2f4" (UID: "6a876178-dc85-41ea-8087-4533f316e2f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:14:08.236525 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.236484    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a876178-dc85-41ea-8087-4533f316e2f4-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:14:08.236525 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.236524    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4qkr\" (UniqueName: \"kubernetes.io/projected/6a876178-dc85-41ea-8087-4533f316e2f4-kube-api-access-v4qkr\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:14:08.236781 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.236540    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a876178-dc85-41ea-8087-4533f316e2f4-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:14:08.236781 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.236554    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a876178-dc85-41ea-8087-4533f316e2f4-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:14:08.656648 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.656556    4900 generic.go:356] "Generic (PLEG): container finished" podID="6a876178-dc85-41ea-8087-4533f316e2f4" containerID="66d5f837a2d49ebd86ce6b24a862cc7ddcd1855e65aacdd1fcb33d58796cae45" exitCode=0
Oct 31 03:14:08.656648 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.656609    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"
Oct 31 03:14:08.656648 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.656626    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw" event={"ID":"6a876178-dc85-41ea-8087-4533f316e2f4","Type":"ContainerDied","Data":"66d5f837a2d49ebd86ce6b24a862cc7ddcd1855e65aacdd1fcb33d58796cae45"}
Oct 31 03:14:08.656961 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.656668    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw" event={"ID":"6a876178-dc85-41ea-8087-4533f316e2f4","Type":"ContainerDied","Data":"2d4630f2e2ea58409aa85e93cec50acbec03170f02707b1efdcea159f358bbfc"}
Oct 31 03:14:08.656961 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.656689    4900 scope.go:122] "RemoveContainer" containerID="66d5f837a2d49ebd86ce6b24a862cc7ddcd1855e65aacdd1fcb33d58796cae45"
Oct 31 03:14:08.668884 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.668860    4900 scope.go:122] "RemoveContainer" containerID="66d5f837a2d49ebd86ce6b24a862cc7ddcd1855e65aacdd1fcb33d58796cae45"
Oct 31 03:14:08.669516 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:14:08.669495    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d5f837a2d49ebd86ce6b24a862cc7ddcd1855e65aacdd1fcb33d58796cae45\": container with ID starting with 66d5f837a2d49ebd86ce6b24a862cc7ddcd1855e65aacdd1fcb33d58796cae45 not found: ID does not exist" containerID="66d5f837a2d49ebd86ce6b24a862cc7ddcd1855e65aacdd1fcb33d58796cae45"
Oct 31 03:14:08.669601 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.669522    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d5f837a2d49ebd86ce6b24a862cc7ddcd1855e65aacdd1fcb33d58796cae45"} err="failed to get container status \"66d5f837a2d49ebd86ce6b24a862cc7ddcd1855e65aacdd1fcb33d58796cae45\": rpc error: code = NotFound desc = could not find container \"66d5f837a2d49ebd86ce6b24a862cc7ddcd1855e65aacdd1fcb33d58796cae45\": container with ID starting with 66d5f837a2d49ebd86ce6b24a862cc7ddcd1855e65aacdd1fcb33d58796cae45 not found: ID does not exist"
Oct 31 03:14:08.808322 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.808287    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"]
Oct 31 03:14:08.854638 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:08.854605    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-586d94f9c6-h8qlw"]
Oct 31 03:14:09.151027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:09.150994    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a876178-dc85-41ea-8087-4533f316e2f4" path="/var/lib/kubelet/pods/6a876178-dc85-41ea-8087-4533f316e2f4/volumes"
Oct 31 03:14:36.983204 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:14:36.983164    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:14:44.025762 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:44.025725    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:14:59.055465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.055430    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8zhq8"]
Oct 31 03:14:59.055985 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.055935    4900 cpu_manager.go:414] "containerMap: removing container" podUID="6a876178-dc85-41ea-8087-4533f316e2f4" containerName="maas-api"
Oct 31 03:14:59.055985 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.055954    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a876178-dc85-41ea-8087-4533f316e2f4" containerName="maas-api"
Oct 31 03:14:59.056101 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.056022    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a876178-dc85-41ea-8087-4533f316e2f4" containerName="maas-api"
Oct 31 03:14:59.058852 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.058831    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8"
Oct 31 03:14:59.107832 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.107806    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 03:14:59.107965 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.107954    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 03:14:59.147745 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.147714    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:14:59.154027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.153992    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 03:14:59.154027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.154021    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-f6lz6\""
Oct 31 03:14:59.154192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.154024    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 03:14:59.155108 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.155092    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8zhq8"]
Oct 31 03:14:59.158478 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.158455    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkdxr\" (UniqueName: \"kubernetes.io/projected/20d71b09-7eb6-434f-b401-44c258927a33-kube-api-access-qkdxr\") pod \"ai-gateway-controller-7d85675f98-8zhq8\" (UID: \"20d71b09-7eb6-434f-b401-44c258927a33\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8"
Oct 31 03:14:59.158569 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.158511    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20d71b09-7eb6-434f-b401-44c258927a33-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-8zhq8\" (UID: \"20d71b09-7eb6-434f-b401-44c258927a33\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8"
Oct 31 03:14:59.259903 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.259871    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkdxr\" (UniqueName: \"kubernetes.io/projected/20d71b09-7eb6-434f-b401-44c258927a33-kube-api-access-qkdxr\") pod \"ai-gateway-controller-7d85675f98-8zhq8\" (UID: \"20d71b09-7eb6-434f-b401-44c258927a33\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8"
Oct 31 03:14:59.260057 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.259924    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20d71b09-7eb6-434f-b401-44c258927a33-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-8zhq8\" (UID: \"20d71b09-7eb6-434f-b401-44c258927a33\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8"
Oct 31 03:14:59.262515 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.262491    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20d71b09-7eb6-434f-b401-44c258927a33-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-8zhq8\" (UID: \"20d71b09-7eb6-434f-b401-44c258927a33\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8"
Oct 31 03:14:59.314060 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.313994    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkdxr\" (UniqueName: \"kubernetes.io/projected/20d71b09-7eb6-434f-b401-44c258927a33-kube-api-access-qkdxr\") pod \"ai-gateway-controller-7d85675f98-8zhq8\" (UID: \"20d71b09-7eb6-434f-b401-44c258927a33\") " pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8"
Oct 31 03:14:59.371061 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.371026    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8"
Oct 31 03:14:59.415286 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.415254    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:14:59.733892 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.733748    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8zhq8"]
Oct 31 03:14:59.736644 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:14:59.736618    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d71b09_7eb6_434f_b401_44c258927a33.slice/crio-1b64ea10508c561285acb1727d46d9e79358f26595e2bab82865083083d0e713 WatchSource:0}: Error finding container 1b64ea10508c561285acb1727d46d9e79358f26595e2bab82865083083d0e713: Status 404 returned error can't find the container with id 1b64ea10508c561285acb1727d46d9e79358f26595e2bab82865083083d0e713
Oct 31 03:14:59.836424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:14:59.836381    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8" event={"ID":"20d71b09-7eb6-434f-b401-44c258927a33","Type":"ContainerStarted","Data":"1b64ea10508c561285acb1727d46d9e79358f26595e2bab82865083083d0e713"}
Oct 31 03:15:04.856649 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:04.856562    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8" event={"ID":"20d71b09-7eb6-434f-b401-44c258927a33","Type":"ContainerStarted","Data":"3f25e24e8342148afaf67be9ccc353139e8eee6e6ea28664c3eb1d451fe610d6"}
Oct 31 03:15:04.857075 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:04.856694    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8"
Oct 31 03:15:04.967219 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:04.967165    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8" podStartSLOduration=1.175000851 podStartE2EDuration="5.96715041s" totalImagesPullingTime="4.792149559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:04.965985715 +0000 UTC m=+457.932319053" watchObservedRunningTime="2026-10-31 03:15:04.96715041 +0000 UTC m=+457.933483754"
Oct 31 03:15:06.983889 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:15:06.983840    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:15:08.594473 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:08.594436    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:15:08.845956 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:08.845856    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:15:08.929765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:08.929728    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"]
Oct 31 03:15:08.932979 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:08.932960    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"
Oct 31 03:15:08.983967 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:08.983931    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-j6fph\""
Oct 31 03:15:09.030441 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.030360    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 03:15:09.031533 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.031505    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"]
Oct 31 03:15:09.042776 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.042737    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj2tw\" (UniqueName: \"kubernetes.io/projected/3158fd77-32bc-4a70-8937-8b58750d82f3-kube-api-access-pj2tw\") pod \"payload-pre-processing-7b98bb678c-9pzbs\" (UID: \"3158fd77-32bc-4a70-8937-8b58750d82f3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"
Oct 31 03:15:09.042954 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.042857    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3158fd77-32bc-4a70-8937-8b58750d82f3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9pzbs\" (UID: \"3158fd77-32bc-4a70-8937-8b58750d82f3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"
Oct 31 03:15:09.077858 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.077819    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"]
Oct 31 03:15:09.081097 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.081078    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"
Oct 31 03:15:09.118187 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.118108    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"]
Oct 31 03:15:09.144237 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.144201    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj2tw\" (UniqueName: \"kubernetes.io/projected/3158fd77-32bc-4a70-8937-8b58750d82f3-kube-api-access-pj2tw\") pod \"payload-pre-processing-7b98bb678c-9pzbs\" (UID: \"3158fd77-32bc-4a70-8937-8b58750d82f3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"
Oct 31 03:15:09.144421 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.144266    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3158fd77-32bc-4a70-8937-8b58750d82f3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9pzbs\" (UID: \"3158fd77-32bc-4a70-8937-8b58750d82f3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"
Oct 31 03:15:09.144421 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.144292    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqpqg\" (UniqueName: \"kubernetes.io/projected/0b2b721d-414f-449f-ba75-918fb3a59972-kube-api-access-wqpqg\") pod \"payload-processing-55fbbbf9dc-k2qfn\" (UID: \"0b2b721d-414f-449f-ba75-918fb3a59972\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"
Oct 31 03:15:09.144421 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.144337    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2b721d-414f-449f-ba75-918fb3a59972-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-k2qfn\" (UID: \"0b2b721d-414f-449f-ba75-918fb3a59972\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"
Oct 31 03:15:09.145109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.145083    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3158fd77-32bc-4a70-8937-8b58750d82f3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9pzbs\" (UID: \"3158fd77-32bc-4a70-8937-8b58750d82f3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"
Oct 31 03:15:09.199032 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.198991    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj2tw\" (UniqueName: \"kubernetes.io/projected/3158fd77-32bc-4a70-8937-8b58750d82f3-kube-api-access-pj2tw\") pod \"payload-pre-processing-7b98bb678c-9pzbs\" (UID: \"3158fd77-32bc-4a70-8937-8b58750d82f3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"
Oct 31 03:15:09.244868 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.244830    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"
Oct 31 03:15:09.245052 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.245028    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2b721d-414f-449f-ba75-918fb3a59972-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-k2qfn\" (UID: \"0b2b721d-414f-449f-ba75-918fb3a59972\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"
Oct 31 03:15:09.245150 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.245134    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqpqg\" (UniqueName: \"kubernetes.io/projected/0b2b721d-414f-449f-ba75-918fb3a59972-kube-api-access-wqpqg\") pod \"payload-processing-55fbbbf9dc-k2qfn\" (UID: \"0b2b721d-414f-449f-ba75-918fb3a59972\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"
Oct 31 03:15:09.245861 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.245844    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2b721d-414f-449f-ba75-918fb3a59972-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-k2qfn\" (UID: \"0b2b721d-414f-449f-ba75-918fb3a59972\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"
Oct 31 03:15:09.303820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.301478    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqpqg\" (UniqueName: \"kubernetes.io/projected/0b2b721d-414f-449f-ba75-918fb3a59972-kube-api-access-wqpqg\") pod \"payload-processing-55fbbbf9dc-k2qfn\" (UID: \"0b2b721d-414f-449f-ba75-918fb3a59972\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"
Oct 31 03:15:09.393514 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.393410    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"
Oct 31 03:15:09.608062 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.608027    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"]
Oct 31 03:15:09.610996 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:15:09.610963    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3158fd77_32bc_4a70_8937_8b58750d82f3.slice/crio-122451269aab11ba5bd050422bf05228021e5e70ce5f8acff6a7837ed3eff8b6 WatchSource:0}: Error finding container 122451269aab11ba5bd050422bf05228021e5e70ce5f8acff6a7837ed3eff8b6: Status 404 returned error can't find the container with id 122451269aab11ba5bd050422bf05228021e5e70ce5f8acff6a7837ed3eff8b6
Oct 31 03:15:09.753964 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.753846    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"]
Oct 31 03:15:09.756878 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:15:09.756849    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2b721d_414f_449f_ba75_918fb3a59972.slice/crio-5f1e9c6f9c150f073b73bebb8aa2949489f508c58d104082ab198da94e226312 WatchSource:0}: Error finding container 5f1e9c6f9c150f073b73bebb8aa2949489f508c58d104082ab198da94e226312: Status 404 returned error can't find the container with id 5f1e9c6f9c150f073b73bebb8aa2949489f508c58d104082ab198da94e226312
Oct 31 03:15:09.875847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.875813    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn" event={"ID":"0b2b721d-414f-449f-ba75-918fb3a59972","Type":"ContainerStarted","Data":"5f1e9c6f9c150f073b73bebb8aa2949489f508c58d104082ab198da94e226312"}
Oct 31 03:15:09.876873 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:09.876847    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs" event={"ID":"3158fd77-32bc-4a70-8937-8b58750d82f3","Type":"ContainerStarted","Data":"122451269aab11ba5bd050422bf05228021e5e70ce5f8acff6a7837ed3eff8b6"}
Oct 31 03:15:12.891065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:12.890955    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn" event={"ID":"0b2b721d-414f-449f-ba75-918fb3a59972","Type":"ContainerStarted","Data":"bc9be7a87afd89e4ba31fe71e5f4c837685247cd3a4e8abe301aa6638b44034b"}
Oct 31 03:15:12.891065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:12.891056    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"
Oct 31 03:15:12.892439 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:12.892416    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs" event={"ID":"3158fd77-32bc-4a70-8937-8b58750d82f3","Type":"ContainerStarted","Data":"81ba4cfe4fa9008c8af963edd30123f97a291f337e2f78691c69d9a6c266a51f"}
Oct 31 03:15:12.892542 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:12.892479    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"
Oct 31 03:15:13.002052 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:13.001999    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn" podStartSLOduration=1.017996983 podStartE2EDuration="4.001984365s" totalImagesPullingTime="2.983987382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:13.001029704 +0000 UTC m=+465.967363047" watchObservedRunningTime="2026-10-31 03:15:13.001984365 +0000 UTC m=+465.968317709"
Oct 31 03:15:13.109610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:13.109557    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs" podStartSLOduration=2.086958114 podStartE2EDuration="5.109543772s" totalImagesPullingTime="3.022585658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:13.108576809 +0000 UTC m=+466.074910154" watchObservedRunningTime="2026-10-31 03:15:13.109543772 +0000 UTC m=+466.075877116"
Oct 31 03:15:15.862743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:15.862713    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8"
Oct 31 03:15:16.205231 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:16.205155    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5cb476b87b-tk2rk"]
Oct 31 03:15:16.208258 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:16.208240    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5cb476b87b-tk2rk"
Oct 31 03:15:16.261756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:16.261725    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5cb476b87b-tk2rk"]
Oct 31 03:15:16.307046 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:16.307009    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b8d483-9270-45ec-94dc-d3e1bc625ca2-metrics-certs\") pod \"ai-gateway-controller-5cb476b87b-tk2rk\" (UID: \"b7b8d483-9270-45ec-94dc-d3e1bc625ca2\") " pod="opendatahub/ai-gateway-controller-5cb476b87b-tk2rk"
Oct 31 03:15:16.307239 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:16.307156    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67l7g\" (UniqueName: \"kubernetes.io/projected/b7b8d483-9270-45ec-94dc-d3e1bc625ca2-kube-api-access-67l7g\") pod \"ai-gateway-controller-5cb476b87b-tk2rk\" (UID: \"b7b8d483-9270-45ec-94dc-d3e1bc625ca2\") " pod="opendatahub/ai-gateway-controller-5cb476b87b-tk2rk"
Oct 31 03:15:16.407596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:16.407558    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67l7g\" (UniqueName: \"kubernetes.io/projected/b7b8d483-9270-45ec-94dc-d3e1bc625ca2-kube-api-access-67l7g\") pod \"ai-gateway-controller-5cb476b87b-tk2rk\" (UID: \"b7b8d483-9270-45ec-94dc-d3e1bc625ca2\") " pod="opendatahub/ai-gateway-controller-5cb476b87b-tk2rk"
Oct 31 03:15:16.407769 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:16.407627    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b8d483-9270-45ec-94dc-d3e1bc625ca2-metrics-certs\") pod \"ai-gateway-controller-5cb476b87b-tk2rk\" (UID: \"b7b8d483-9270-45ec-94dc-d3e1bc625ca2\") " pod="opendatahub/ai-gateway-controller-5cb476b87b-tk2rk"
Oct 31 03:15:16.410011 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:16.409985    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b8d483-9270-45ec-94dc-d3e1bc625ca2-metrics-certs\") pod \"ai-gateway-controller-5cb476b87b-tk2rk\" (UID: \"b7b8d483-9270-45ec-94dc-d3e1bc625ca2\") " pod="opendatahub/ai-gateway-controller-5cb476b87b-tk2rk"
Oct 31 03:15:16.461141 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:16.461072    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67l7g\" (UniqueName: \"kubernetes.io/projected/b7b8d483-9270-45ec-94dc-d3e1bc625ca2-kube-api-access-67l7g\") pod \"ai-gateway-controller-5cb476b87b-tk2rk\" (UID: \"b7b8d483-9270-45ec-94dc-d3e1bc625ca2\") " pod="opendatahub/ai-gateway-controller-5cb476b87b-tk2rk"
Oct 31 03:15:16.520118 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:16.520077    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5cb476b87b-tk2rk"
Oct 31 03:15:16.880925 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:16.880833    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5cb476b87b-tk2rk"]
Oct 31 03:15:16.883546 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:15:16.883517    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b8d483_9270_45ec_94dc_d3e1bc625ca2.slice/crio-9dbde08e8f0e767c29c30895952017c71f7194623bffd2ec047bacf046b001e0 WatchSource:0}: Error finding container 9dbde08e8f0e767c29c30895952017c71f7194623bffd2ec047bacf046b001e0: Status 404 returned error can't find the container with id 9dbde08e8f0e767c29c30895952017c71f7194623bffd2ec047bacf046b001e0
Oct 31 03:15:16.908905 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:16.908875    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5cb476b87b-tk2rk" event={"ID":"b7b8d483-9270-45ec-94dc-d3e1bc625ca2","Type":"ContainerStarted","Data":"9dbde08e8f0e767c29c30895952017c71f7194623bffd2ec047bacf046b001e0"}
Oct 31 03:15:17.914084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:17.914045    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5cb476b87b-tk2rk" event={"ID":"b7b8d483-9270-45ec-94dc-d3e1bc625ca2","Type":"ContainerStarted","Data":"1aec528b68afeaf99648cf890085758387e513b12c4d1e04ec6a82e031d0ecb2"}
Oct 31 03:15:17.914481 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:17.914213    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5cb476b87b-tk2rk"
Oct 31 03:15:23.899092 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:23.899062    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"
Oct 31 03:15:23.899483 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:23.899131    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"
Oct 31 03:15:24.008058 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:24.008009    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5cb476b87b-tk2rk" podStartSLOduration=7.073564567 podStartE2EDuration="8.00799563s" totalImagesPullingTime="934.431063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:18.021649914 +0000 UTC m=+470.987983259" watchObservedRunningTime="2026-10-31 03:15:24.00799563 +0000 UTC m=+476.974328973"
Oct 31 03:15:28.921699 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:28.921669    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5cb476b87b-tk2rk"
Oct 31 03:15:29.076368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.076334    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8zhq8"]
Oct 31 03:15:29.076615 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.076589    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8" podUID="20d71b09-7eb6-434f-b401-44c258927a33" containerName="manager" containerID="cri-o://3f25e24e8342148afaf67be9ccc353139e8eee6e6ea28664c3eb1d451fe610d6" gracePeriod=30
Oct 31 03:15:29.319561 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.319538    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8"
Oct 31 03:15:29.420832 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.420801    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20d71b09-7eb6-434f-b401-44c258927a33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20d71b09-7eb6-434f-b401-44c258927a33-metrics-certs\") pod \"20d71b09-7eb6-434f-b401-44c258927a33\" (UID: \"20d71b09-7eb6-434f-b401-44c258927a33\") "
Oct 31 03:15:29.420992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.420920    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d71b09-7eb6-434f-b401-44c258927a33-kube-api-access-qkdxr\" (UniqueName: \"kubernetes.io/projected/20d71b09-7eb6-434f-b401-44c258927a33-kube-api-access-qkdxr\") pod \"20d71b09-7eb6-434f-b401-44c258927a33\" (UID: \"20d71b09-7eb6-434f-b401-44c258927a33\") "
Oct 31 03:15:29.422946 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.422920    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d71b09-7eb6-434f-b401-44c258927a33-kube-api-access-qkdxr" pod "20d71b09-7eb6-434f-b401-44c258927a33" (UID: "20d71b09-7eb6-434f-b401-44c258927a33"). InnerVolumeSpecName "kube-api-access-qkdxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:15:29.423066 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.422922    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d71b09-7eb6-434f-b401-44c258927a33-metrics-certs" pod "20d71b09-7eb6-434f-b401-44c258927a33" (UID: "20d71b09-7eb6-434f-b401-44c258927a33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:15:29.521981 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.521944    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20d71b09-7eb6-434f-b401-44c258927a33-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:15:29.521981 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.521984    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkdxr\" (UniqueName: \"kubernetes.io/projected/20d71b09-7eb6-434f-b401-44c258927a33-kube-api-access-qkdxr\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:15:29.963101 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.963017    4900 generic.go:356] "Generic (PLEG): container finished" podID="20d71b09-7eb6-434f-b401-44c258927a33" containerID="3f25e24e8342148afaf67be9ccc353139e8eee6e6ea28664c3eb1d451fe610d6" exitCode=0
Oct 31 03:15:29.963101 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.963081    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8"
Oct 31 03:15:29.963513 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.963082    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8" event={"ID":"20d71b09-7eb6-434f-b401-44c258927a33","Type":"ContainerDied","Data":"3f25e24e8342148afaf67be9ccc353139e8eee6e6ea28664c3eb1d451fe610d6"}
Oct 31 03:15:29.963513 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.963225    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-8zhq8" event={"ID":"20d71b09-7eb6-434f-b401-44c258927a33","Type":"ContainerDied","Data":"1b64ea10508c561285acb1727d46d9e79358f26595e2bab82865083083d0e713"}
Oct 31 03:15:29.963513 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.963251    4900 scope.go:122] "RemoveContainer" containerID="3f25e24e8342148afaf67be9ccc353139e8eee6e6ea28664c3eb1d451fe610d6"
Oct 31 03:15:29.971863 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.971842    4900 scope.go:122] "RemoveContainer" containerID="3f25e24e8342148afaf67be9ccc353139e8eee6e6ea28664c3eb1d451fe610d6"
Oct 31 03:15:29.972143 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:15:29.972122    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f25e24e8342148afaf67be9ccc353139e8eee6e6ea28664c3eb1d451fe610d6\": container with ID starting with 3f25e24e8342148afaf67be9ccc353139e8eee6e6ea28664c3eb1d451fe610d6 not found: ID does not exist" containerID="3f25e24e8342148afaf67be9ccc353139e8eee6e6ea28664c3eb1d451fe610d6"
Oct 31 03:15:29.972185 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:29.972159    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f25e24e8342148afaf67be9ccc353139e8eee6e6ea28664c3eb1d451fe610d6"} err="failed to get container status \"3f25e24e8342148afaf67be9ccc353139e8eee6e6ea28664c3eb1d451fe610d6\": rpc error: code = NotFound desc = could not find container \"3f25e24e8342148afaf67be9ccc353139e8eee6e6ea28664c3eb1d451fe610d6\": container with ID starting with 3f25e24e8342148afaf67be9ccc353139e8eee6e6ea28664c3eb1d451fe610d6 not found: ID does not exist"
Oct 31 03:15:30.115066 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:30.115028    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8zhq8"]
Oct 31 03:15:30.161795 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:30.161764    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-8zhq8"]
Oct 31 03:15:30.767765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:30.767734    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:15:31.150769 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:31.150691    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d71b09-7eb6-434f-b401-44c258927a33" path="/var/lib/kubelet/pods/20d71b09-7eb6-434f-b401-44c258927a33/volumes"
Oct 31 03:15:31.214334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:31.214291    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:15:32.448669 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:32.448635    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:15:32.681259 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:32.681228    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:15:36.982004 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:15:36.981967    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:15:38.723394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.723359    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"]
Oct 31 03:15:38.723823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.723683    4900 cpu_manager.go:414] "containerMap: removing container" podUID="20d71b09-7eb6-434f-b401-44c258927a33" containerName="manager"
Oct 31 03:15:38.723823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.723693    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d71b09-7eb6-434f-b401-44c258927a33" containerName="manager"
Oct 31 03:15:38.723823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.723769    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d71b09-7eb6-434f-b401-44c258927a33" containerName="manager"
Oct 31 03:15:38.729752 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.729734    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.780779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.780736    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 03:15:38.800048 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.800007    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpnz9\" (UniqueName: \"kubernetes.io/projected/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-kube-api-access-qpnz9\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.800218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.800061    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.800218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.800119    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.800218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.800195    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.800353 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.800261    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.800353 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.800299    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.826540 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.826504    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 03:15:38.826540 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.826529    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-trl42\""
Oct 31 03:15:38.826716 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.826608    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 31 03:15:38.827683 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.827664    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"]
Oct 31 03:15:38.901626 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.901588    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpnz9\" (UniqueName: \"kubernetes.io/projected/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-kube-api-access-qpnz9\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.901626 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.901623    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.901897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.901640    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.901897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.901662    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.901897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.901708    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.901897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.901758    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.902092 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.902069    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.902154 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.902131    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.902191 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.902144    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.904029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.904009    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.904223 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.904205    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:38.907120 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.907097    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:15:38.957479 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:38.957448    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpnz9\" (UniqueName: \"kubernetes.io/projected/42af8fc7-c7bd-4af5-b122-5dec2cd2dd85-kube-api-access-qpnz9\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv\" (UID: \"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:39.042167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:39.042128    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:39.405875 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:39.405744    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"]
Oct 31 03:15:39.408656 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:15:39.408629    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42af8fc7_c7bd_4af5_b122_5dec2cd2dd85.slice/crio-3a19c62856fe264168f0ab570bcf6348244aa371db72e23e8adf5280c2445a05 WatchSource:0}: Error finding container 3a19c62856fe264168f0ab570bcf6348244aa371db72e23e8adf5280c2445a05: Status 404 returned error can't find the container with id 3a19c62856fe264168f0ab570bcf6348244aa371db72e23e8adf5280c2445a05
Oct 31 03:15:40.004465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:40.004427    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv" event={"ID":"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85","Type":"ContainerStarted","Data":"3a19c62856fe264168f0ab570bcf6348244aa371db72e23e8adf5280c2445a05"}
Oct 31 03:15:42.013850 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:42.013814    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv" event={"ID":"42af8fc7-c7bd-4af5-b122-5dec2cd2dd85","Type":"ContainerStarted","Data":"97e8b882f9d44d23b867d267c11250c4277f3e9027bedcfdad0c79d63f7d5260"}
Oct 31 03:15:42.043346 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:42.043314    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:42.128972 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:42.125284    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv" podStartSLOduration=1.880260202 podStartE2EDuration="4.125266213s" totalImagesPullingTime="2.245006011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:42.123027514 +0000 UTC m=+495.089360861" watchObservedRunningTime="2026-10-31 03:15:42.125266213 +0000 UTC m=+495.091599558"
Oct 31 03:15:44.361222 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:44.361167    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:15:45.762599 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.762564    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"]
Oct 31 03:15:45.766018 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.766000    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.860843 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.860810    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 31 03:15:45.862043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.862020    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"]
Oct 31 03:15:45.863470 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.863454    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkzg5\" (UniqueName: \"kubernetes.io/projected/520f2099-f186-4222-a31a-9f073c33dced-kube-api-access-fkzg5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.863530 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.863480    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/520f2099-f186-4222-a31a-9f073c33dced-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.863530 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.863504    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/520f2099-f186-4222-a31a-9f073c33dced-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.863602 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.863585    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/520f2099-f186-4222-a31a-9f073c33dced-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.863636 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.863611    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/520f2099-f186-4222-a31a-9f073c33dced-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.863694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.863679    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/520f2099-f186-4222-a31a-9f073c33dced-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.957538 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.957503    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:15:45.964685 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.964658    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/520f2099-f186-4222-a31a-9f073c33dced-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.964779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.964690    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/520f2099-f186-4222-a31a-9f073c33dced-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.964779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.964707    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/520f2099-f186-4222-a31a-9f073c33dced-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.964869 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.964809    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkzg5\" (UniqueName: \"kubernetes.io/projected/520f2099-f186-4222-a31a-9f073c33dced-kube-api-access-fkzg5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.964869 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.964841    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/520f2099-f186-4222-a31a-9f073c33dced-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.964971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.964882    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/520f2099-f186-4222-a31a-9f073c33dced-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.965153 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.965131    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/520f2099-f186-4222-a31a-9f073c33dced-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.965232 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.965179    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/520f2099-f186-4222-a31a-9f073c33dced-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.965270 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.965244    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/520f2099-f186-4222-a31a-9f073c33dced-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.966955 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.966937    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/520f2099-f186-4222-a31a-9f073c33dced-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:45.967208 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:45.967191    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/520f2099-f186-4222-a31a-9f073c33dced-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:46.022765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:46.022686    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkzg5\" (UniqueName: \"kubernetes.io/projected/520f2099-f186-4222-a31a-9f073c33dced-kube-api-access-fkzg5\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-n42bp\" (UID: \"520f2099-f186-4222-a31a-9f073c33dced\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:46.078421 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:46.078385    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:46.439975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:46.439950    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"]
Oct 31 03:15:46.442381 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:15:46.442350    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520f2099_f186_4222_a31a_9f073c33dced.slice/crio-3400d66ed48cb0518d1d28cac70acde45fa25e9021bbb49e4b48d5fd73f6458e WatchSource:0}: Error finding container 3400d66ed48cb0518d1d28cac70acde45fa25e9021bbb49e4b48d5fd73f6458e: Status 404 returned error can't find the container with id 3400d66ed48cb0518d1d28cac70acde45fa25e9021bbb49e4b48d5fd73f6458e
Oct 31 03:15:47.036332 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:47.036296    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp" event={"ID":"520f2099-f186-4222-a31a-9f073c33dced","Type":"ContainerStarted","Data":"3400d66ed48cb0518d1d28cac70acde45fa25e9021bbb49e4b48d5fd73f6458e"}
Oct 31 03:15:48.041667 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:48.041626    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp" event={"ID":"520f2099-f186-4222-a31a-9f073c33dced","Type":"ContainerStarted","Data":"99d14a9731b65dc049c2f452b2888f69ba0f94add4050883cd3ded8ade8fe729"}
Oct 31 03:15:48.079429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:48.079396    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:48.150597 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:48.150536    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp" podStartSLOduration=2.548338007 podStartE2EDuration="3.150520181s" totalImagesPullingTime="602.182174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:48.149181066 +0000 UTC m=+501.115514412" watchObservedRunningTime="2026-10-31 03:15:48.150520181 +0000 UTC m=+501.116853544"
Oct 31 03:15:49.043147 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.043112    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:49.057291 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.057268    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:49.391051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.390974    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"]
Oct 31 03:15:49.394349 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.394332    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.443101 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.443069    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 31 03:15:49.448465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.448439    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"]
Oct 31 03:15:49.494433 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.494404    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.494595 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.494458    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.494595 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.494556    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.494595 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.494587    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.494733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.494604    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdn6m\" (UniqueName: \"kubernetes.io/projected/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-kube-api-access-sdn6m\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.494733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.494722    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.572796 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.572754    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:15:49.595285 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.595256    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.595390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.595295    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.595390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.595321    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.595390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.595338    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.595390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.595353    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdn6m\" (UniqueName: \"kubernetes.io/projected/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-kube-api-access-sdn6m\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.595390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.595384    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.595734 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.595712    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.595820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.595741    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.595820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.595757    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.597623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.597595    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.597891 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.597873    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.648491 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.648426    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdn6m\" (UniqueName: \"kubernetes.io/projected/3792ff36-75f6-4fb5-b67b-ee4f086c6ef3-kube-api-access-sdn6m\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-psl4l\" (UID: \"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:49.706589 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:49.706548    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:50.054623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:50.054596    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv"
Oct 31 03:15:50.068457 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:50.068434    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"]
Oct 31 03:15:50.070905 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:15:50.070883    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3792ff36_75f6_4fb5_b67b_ee4f086c6ef3.slice/crio-993898790ec2e3acff90d84b6c92762e4da7048908a7e937856f744707ff8aa2 WatchSource:0}: Error finding container 993898790ec2e3acff90d84b6c92762e4da7048908a7e937856f744707ff8aa2: Status 404 returned error can't find the container with id 993898790ec2e3acff90d84b6c92762e4da7048908a7e937856f744707ff8aa2
Oct 31 03:15:51.056952 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:51.056911    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l" event={"ID":"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3","Type":"ContainerStarted","Data":"077a1df687fabe233505ddefff1564832387c79dcb41eb9d1723098c05ef94ff"}
Oct 31 03:15:51.057355 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:51.056994    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l" event={"ID":"3792ff36-75f6-4fb5-b67b-ee4f086c6ef3","Type":"ContainerStarted","Data":"993898790ec2e3acff90d84b6c92762e4da7048908a7e937856f744707ff8aa2"}
Oct 31 03:15:51.707624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:51.707589    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:52.931445 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:52.931386    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l" podStartSLOduration=3.325735903 podStartE2EDuration="3.93136979s" totalImagesPullingTime="605.633887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:51.166688224 +0000 UTC m=+504.133021572" watchObservedRunningTime="2026-10-31 03:15:52.93136979 +0000 UTC m=+505.897703155"
Oct 31 03:15:52.932037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:52.932021    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"]
Oct 31 03:15:52.935484 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:52.935467    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.026048 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.026012    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4f0318-9063-4c10-8c6a-240ed741b154-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.026192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.026060    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hw29\" (UniqueName: \"kubernetes.io/projected/2f4f0318-9063-4c10-8c6a-240ed741b154-kube-api-access-9hw29\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.026192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.026114    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4f0318-9063-4c10-8c6a-240ed741b154-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.026192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.026170    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4f0318-9063-4c10-8c6a-240ed741b154-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.026295 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.026202    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4f0318-9063-4c10-8c6a-240ed741b154-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.026295 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.026224    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4f0318-9063-4c10-8c6a-240ed741b154-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.030268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.030250    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 31 03:15:53.031571 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.031553    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"]
Oct 31 03:15:53.095217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.095186    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:15:53.127460 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.127437    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4f0318-9063-4c10-8c6a-240ed741b154-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.127605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.127469    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4f0318-9063-4c10-8c6a-240ed741b154-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.127605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.127546    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4f0318-9063-4c10-8c6a-240ed741b154-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.127605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.127570    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hw29\" (UniqueName: \"kubernetes.io/projected/2f4f0318-9063-4c10-8c6a-240ed741b154-kube-api-access-9hw29\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.127605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.127593    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4f0318-9063-4c10-8c6a-240ed741b154-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.127836 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.127636    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4f0318-9063-4c10-8c6a-240ed741b154-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.128012 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.127988    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4f0318-9063-4c10-8c6a-240ed741b154-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.128135 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.128062    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4f0318-9063-4c10-8c6a-240ed741b154-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.128181 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.128134    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4f0318-9063-4c10-8c6a-240ed741b154-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.129880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.129859    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4f0318-9063-4c10-8c6a-240ed741b154-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.129992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.129977    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4f0318-9063-4c10-8c6a-240ed741b154-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.184926 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.184859    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hw29\" (UniqueName: \"kubernetes.io/projected/2f4f0318-9063-4c10-8c6a-240ed741b154-kube-api-access-9hw29\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-4mdxf\" (UID: \"2f4f0318-9063-4c10-8c6a-240ed741b154\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.247523 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.247483    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:53.610142 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:53.610106    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"]
Oct 31 03:15:53.613139 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:15:53.613109    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4f0318_9063_4c10_8c6a_240ed741b154.slice/crio-6b6e8be830412e085f2ab967117592a98ce0ac1e29cfcf244e7f0a56d6c4c543 WatchSource:0}: Error finding container 6b6e8be830412e085f2ab967117592a98ce0ac1e29cfcf244e7f0a56d6c4c543: Status 404 returned error can't find the container with id 6b6e8be830412e085f2ab967117592a98ce0ac1e29cfcf244e7f0a56d6c4c543
Oct 31 03:15:54.072572 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:54.072537    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf" event={"ID":"2f4f0318-9063-4c10-8c6a-240ed741b154","Type":"ContainerStarted","Data":"6b6e8be830412e085f2ab967117592a98ce0ac1e29cfcf244e7f0a56d6c4c543"}
Oct 31 03:15:55.079339 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:55.079304    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf" event={"ID":"2f4f0318-9063-4c10-8c6a-240ed741b154","Type":"ContainerStarted","Data":"f45fce7f8547d05e1e52528d05dca3175ef48b249ba226e541d2e52462ab92dd"}
Oct 31 03:15:55.191315 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:55.191268    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf" podStartSLOduration=2.558510575 podStartE2EDuration="3.191250319s" totalImagesPullingTime="632.739744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:55.189731702 +0000 UTC m=+508.156065057" watchObservedRunningTime="2026-10-31 03:15:55.191250319 +0000 UTC m=+508.157583663"
Oct 31 03:15:55.248190 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:55.248156    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:15:56.079182 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:56.079147    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:56.091734 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:56.091708    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:56.916418 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:56.916382    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"]
Oct 31 03:15:56.919797 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:56.919772    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:56.968476 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:56.968439    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 31 03:15:57.015876 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.015845    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"]
Oct 31 03:15:57.064145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.064111    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.064309 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.064155    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.064309 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.064261    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjsbc\" (UniqueName: \"kubernetes.io/projected/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-kube-api-access-jjsbc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.064389 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.064366    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.064447 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.064431    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.064500 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.064484    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.086583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.086550    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:15:57.090823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.090800    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-n42bp"
Oct 31 03:15:57.165067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.165034    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.165538 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.165087    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.165538 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.165134    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.165538 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.165181    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.165538 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.165233    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.165538 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.165314    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjsbc\" (UniqueName: \"kubernetes.io/projected/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-kube-api-access-jjsbc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.165853 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.165573    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.165853 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.165590    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.165853 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.165711    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.167442 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.167390    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.167707 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.167690    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.220829 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.220769    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjsbc\" (UniqueName: \"kubernetes.io/projected/37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8-kube-api-access-jjsbc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5\" (UID: \"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.232100 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.232069    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:57.593287 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:57.593262    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"]
Oct 31 03:15:57.595763 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:15:57.595734    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37936d51_1fa0_49f0_9d91_f4f3d0e4a0e8.slice/crio-25293e0720f29e46a5de003ebe265f18c620bdbba8288848dfd06034962c02c2 WatchSource:0}: Error finding container 25293e0720f29e46a5de003ebe265f18c620bdbba8288848dfd06034962c02c2: Status 404 returned error can't find the container with id 25293e0720f29e46a5de003ebe265f18c620bdbba8288848dfd06034962c02c2
Oct 31 03:15:58.094323 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:58.094265    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5" event={"ID":"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8","Type":"ContainerStarted","Data":"25293e0720f29e46a5de003ebe265f18c620bdbba8288848dfd06034962c02c2"}
Oct 31 03:15:59.100631 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:59.100597    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5" event={"ID":"37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8","Type":"ContainerStarted","Data":"46b2c77ffd7856dd640332622f3565402dd8da40a2d3494978a0247d1f109d3d"}
Oct 31 03:15:59.210979 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:59.210920    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5" podStartSLOduration=2.589197613 podStartE2EDuration="3.210903437s" totalImagesPullingTime="621.705824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:59.210242738 +0000 UTC m=+512.176576082" watchObservedRunningTime="2026-10-31 03:15:59.210903437 +0000 UTC m=+512.177236778"
Oct 31 03:15:59.232379 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:59.232349    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:15:59.706982 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:59.706945    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:15:59.719343 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:15:59.719315    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:16:00.116552 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.116527    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-psl4l"
Oct 31 03:16:00.129398 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.129373    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ns4fr"]
Oct 31 03:16:00.132863 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.132847    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.181418 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.181379    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 31 03:16:00.186886 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.186848    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ns4fr"]
Oct 31 03:16:00.191869 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.191846    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/246cfb6b-4be8-4f89-bcc4-58f24acf2229-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.191976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.191877    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/246cfb6b-4be8-4f89-bcc4-58f24acf2229-home\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.191976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.191893    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/246cfb6b-4be8-4f89-bcc4-58f24acf2229-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.191976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.191916    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/246cfb6b-4be8-4f89-bcc4-58f24acf2229-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.192188 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.192172    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/246cfb6b-4be8-4f89-bcc4-58f24acf2229-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.192238 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.192202    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kmlr\" (UniqueName: \"kubernetes.io/projected/246cfb6b-4be8-4f89-bcc4-58f24acf2229-kube-api-access-2kmlr\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.293077 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.293048    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/246cfb6b-4be8-4f89-bcc4-58f24acf2229-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.293077 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.293081    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/246cfb6b-4be8-4f89-bcc4-58f24acf2229-home\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.293316 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.293097    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/246cfb6b-4be8-4f89-bcc4-58f24acf2229-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.293316 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.293129    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/246cfb6b-4be8-4f89-bcc4-58f24acf2229-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.293316 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.293200    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/246cfb6b-4be8-4f89-bcc4-58f24acf2229-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.293316 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.293234    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kmlr\" (UniqueName: \"kubernetes.io/projected/246cfb6b-4be8-4f89-bcc4-58f24acf2229-kube-api-access-2kmlr\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.293528 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.293502    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/246cfb6b-4be8-4f89-bcc4-58f24acf2229-home\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.293619 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.293598    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/246cfb6b-4be8-4f89-bcc4-58f24acf2229-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.293679 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.293624    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/246cfb6b-4be8-4f89-bcc4-58f24acf2229-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.294733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.294696    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:16:00.295480 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.295452    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/246cfb6b-4be8-4f89-bcc4-58f24acf2229-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.295690 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.295670    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/246cfb6b-4be8-4f89-bcc4-58f24acf2229-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.347103 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.347070    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kmlr\" (UniqueName: \"kubernetes.io/projected/246cfb6b-4be8-4f89-bcc4-58f24acf2229-kube-api-access-2kmlr\") pod \"echo-simulator-kserve-64b6f8f96c-ns4fr\" (UID: \"246cfb6b-4be8-4f89-bcc4-58f24acf2229\") " pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.445192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.445121    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:00.807691 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:00.807661    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-ns4fr"]
Oct 31 03:16:00.810283 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:16:00.810256    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246cfb6b_4be8_4f89_bcc4_58f24acf2229.slice/crio-0b7aa7c6ba73970a7c5b460a28b92c9ebaf5277421029ecc5b6bd6d879eda6db WatchSource:0}: Error finding container 0b7aa7c6ba73970a7c5b460a28b92c9ebaf5277421029ecc5b6bd6d879eda6db: Status 404 returned error can't find the container with id 0b7aa7c6ba73970a7c5b460a28b92c9ebaf5277421029ecc5b6bd6d879eda6db
Oct 31 03:16:01.110896 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:01.110807    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr" event={"ID":"246cfb6b-4be8-4f89-bcc4-58f24acf2229","Type":"ContainerStarted","Data":"0b7aa7c6ba73970a7c5b460a28b92c9ebaf5277421029ecc5b6bd6d879eda6db"}
Oct 31 03:16:03.203633 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:03.203592    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:16:03.248495 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:03.248464    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:16:03.261378 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:03.261348    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:16:04.129177 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:04.129142    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr" event={"ID":"246cfb6b-4be8-4f89-bcc4-58f24acf2229","Type":"ContainerStarted","Data":"471ac3bd7d656723df42b05c8fc7fba1598b40a2dae71ae1e17d86b077d8d863"}
Oct 31 03:16:04.140528 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:04.140498    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-4mdxf"
Oct 31 03:16:04.237577 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:04.237521    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr" podStartSLOduration=1.7607177040000002 podStartE2EDuration="4.237505034s" totalImagesPullingTime="2.47678733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:04.236952116 +0000 UTC m=+517.203285461" watchObservedRunningTime="2026-10-31 03:16:04.237505034 +0000 UTC m=+517.203838395"
Oct 31 03:16:04.446046 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:04.445971    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:06.982820 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:16:06.982766    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:16:07.232382 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:07.232346    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:16:07.245087 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:07.245018    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:16:07.848421 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:07.848383    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"]
Oct 31 03:16:07.851892 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:07.851869    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:07.900443 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:07.900414    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 31 03:16:07.947986 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:07.947946    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"]
Oct 31 03:16:07.962625 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:07.962593    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43738834-604f-4063-8c19-d210e1a44de7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:07.962625 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:07.962623    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r52j\" (UniqueName: \"kubernetes.io/projected/43738834-604f-4063-8c19-d210e1a44de7-kube-api-access-7r52j\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:07.962839 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:07.962647    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43738834-604f-4063-8c19-d210e1a44de7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:07.962890 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:07.962836    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43738834-604f-4063-8c19-d210e1a44de7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:07.962925 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:07.962901    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43738834-604f-4063-8c19-d210e1a44de7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:07.962996 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:07.962972    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43738834-604f-4063-8c19-d210e1a44de7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:08.033655 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.033614    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:16:08.063736 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.063703    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43738834-604f-4063-8c19-d210e1a44de7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:08.063889 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.063764    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43738834-604f-4063-8c19-d210e1a44de7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:08.063889 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.063821    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43738834-604f-4063-8c19-d210e1a44de7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:08.063889 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.063860    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43738834-604f-4063-8c19-d210e1a44de7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:08.064012 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.063989    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r52j\" (UniqueName: \"kubernetes.io/projected/43738834-604f-4063-8c19-d210e1a44de7-kube-api-access-7r52j\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:08.064115 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.064031    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43738834-604f-4063-8c19-d210e1a44de7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:08.064285 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.064253    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43738834-604f-4063-8c19-d210e1a44de7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:08.064370 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.064291    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43738834-604f-4063-8c19-d210e1a44de7-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:08.064455 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.064422    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43738834-604f-4063-8c19-d210e1a44de7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:08.066104 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.066076    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43738834-604f-4063-8c19-d210e1a44de7-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:08.066496 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.066475    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43738834-604f-4063-8c19-d210e1a44de7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:08.117885 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.117809    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r52j\" (UniqueName: \"kubernetes.io/projected/43738834-604f-4063-8c19-d210e1a44de7-kube-api-access-7r52j\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j\" (UID: \"43738834-604f-4063-8c19-d210e1a44de7\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:08.158698 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.158666    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5"
Oct 31 03:16:08.170322 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.170296    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:08.535405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:08.535368    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"]
Oct 31 03:16:08.538360 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:16:08.538335    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43738834_604f_4063_8c19_d210e1a44de7.slice/crio-64efd9a20be44ae8f575a6be67e0f6756ea5576ca05c8c9ee8815509914ffc90 WatchSource:0}: Error finding container 64efd9a20be44ae8f575a6be67e0f6756ea5576ca05c8c9ee8815509914ffc90: Status 404 returned error can't find the container with id 64efd9a20be44ae8f575a6be67e0f6756ea5576ca05c8c9ee8815509914ffc90
Oct 31 03:16:09.152990 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:09.152953    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j" event={"ID":"43738834-604f-4063-8c19-d210e1a44de7","Type":"ContainerStarted","Data":"64efd9a20be44ae8f575a6be67e0f6756ea5576ca05c8c9ee8815509914ffc90"}
Oct 31 03:16:10.159100 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:10.159062    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j" event={"ID":"43738834-604f-4063-8c19-d210e1a44de7","Type":"ContainerStarted","Data":"00a5f19cf0c2e43bc26ce03bba5a4ae0cc8ac6d028ffeeb77a6771530caacb44"}
Oct 31 03:16:10.171151 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:10.171119    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:10.267845 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:10.267769    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j" podStartSLOduration=2.652684472 podStartE2EDuration="3.267753701s" totalImagesPullingTime="615.069229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:10.266127408 +0000 UTC m=+523.232460743" watchObservedRunningTime="2026-10-31 03:16:10.267753701 +0000 UTC m=+523.234087046"
Oct 31 03:16:10.446178 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:10.446092    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:10.458916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:10.458891    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:11.174111 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:11.174079    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-ns4fr"
Oct 31 03:16:18.171126 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:18.171091    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:18.183637 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:18.183614    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:18.205371 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:18.205341    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:16:20.240266 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.240229    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"]
Oct 31 03:16:20.246374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.246338    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.282624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.282587    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.282823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.282656    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.282823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.282722    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baed435c-cff8-4c33-b642-a1fd39d437f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.282948 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.282827    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.282948 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.282854    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qf9m\" (UniqueName: \"kubernetes.io/projected/baed435c-cff8-4c33-b642-a1fd39d437f9-kube-api-access-8qf9m\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.282948 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.282893    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.283049 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.282978    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.283049 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.283014    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.283049 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.283034    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.295126 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.295096    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 03:16:20.345270 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.345234    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"]
Oct 31 03:16:20.345603 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.345528    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-96js4\""
Oct 31 03:16:20.384172 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.384143    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.384340 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.384197    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.384340 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.384223    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baed435c-cff8-4c33-b642-a1fd39d437f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.384340 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.384253    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.384340 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.384270    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qf9m\" (UniqueName: \"kubernetes.io/projected/baed435c-cff8-4c33-b642-a1fd39d437f9-kube-api-access-8qf9m\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.384571 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.384539    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.384646 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.384606    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.384646 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.384616    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.384750 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.384704    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.384887 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.384859    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.384971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.384937    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.385032 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.385017    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.385076 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.385038    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baed435c-cff8-4c33-b642-a1fd39d437f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.385076 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.385065    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.386586 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.386561    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.386823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.386777    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.437693 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.437665    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.437932 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.437905    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qf9m\" (UniqueName: \"kubernetes.io/projected/baed435c-cff8-4c33-b642-a1fd39d437f9-kube-api-access-8qf9m\") pod \"maas-default-gateway-openshift-default-6886b96f9-pflrt\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.559029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.558936    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:20.969957 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:20.969925    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"]
Oct 31 03:16:20.973918 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:16:20.973872    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaed435c_cff8_4c33_b642_a1fd39d437f9.slice/crio-b8ed34d86cf92d2ee5847d7204b527ddb0c92429e7ef338c27c7b5a65afabc83 WatchSource:0}: Error finding container b8ed34d86cf92d2ee5847d7204b527ddb0c92429e7ef338c27c7b5a65afabc83: Status 404 returned error can't find the container with id b8ed34d86cf92d2ee5847d7204b527ddb0c92429e7ef338c27c7b5a65afabc83
Oct 31 03:16:21.208397 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:21.208364    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" event={"ID":"baed435c-cff8-4c33-b642-a1fd39d437f9","Type":"ContainerStarted","Data":"b8ed34d86cf92d2ee5847d7204b527ddb0c92429e7ef338c27c7b5a65afabc83"}
Oct 31 03:16:25.846259 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:25.846227    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:16:25.846559 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:25.846299    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:16:25.846559 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:25.846325    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:16:26.231051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:26.230951    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" event={"ID":"baed435c-cff8-4c33-b642-a1fd39d437f9","Type":"ContainerStarted","Data":"45d0fc2c1bae0e7bf8bf05e21b0e9a822c7a4dd01a6c330d381c143e8fcffa7d"}
Oct 31 03:16:26.388029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:26.387972    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" podStartSLOduration=1.519434114 podStartE2EDuration="6.387955684s" totalImagesPullingTime="4.86852157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:26.386363533 +0000 UTC m=+539.352696924" watchObservedRunningTime="2026-10-31 03:16:26.387955684 +0000 UTC m=+539.354289089"
Oct 31 03:16:26.560091 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:26.560051    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:26.561850 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:26.561820    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pflrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 03:16:26.561987 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:26.561882    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" podUID="baed435c-cff8-4c33-b642-a1fd39d437f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 03:16:27.560377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:27.560334    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pflrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 03:16:27.560775 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:27.560412    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" podUID="baed435c-cff8-4c33-b642-a1fd39d437f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 03:16:28.560397 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:28.560361    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pflrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 03:16:28.560815 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:28.560418    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" podUID="baed435c-cff8-4c33-b642-a1fd39d437f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 03:16:29.560303 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:29.560261    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pflrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 03:16:29.560518 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:29.560319    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" podUID="baed435c-cff8-4c33-b642-a1fd39d437f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 03:16:30.560398 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:30.560353    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pflrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 03:16:30.560595 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:30.560429    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" podUID="baed435c-cff8-4c33-b642-a1fd39d437f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 03:16:31.560406 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:31.560360    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pflrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 03:16:31.560590 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:31.560425    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" podUID="baed435c-cff8-4c33-b642-a1fd39d437f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 03:16:32.560050 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:32.560000    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pflrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 31 03:16:32.560468 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:32.560077    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" podUID="baed435c-cff8-4c33-b642-a1fd39d437f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 31 03:16:34.144056 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:34.144021    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:34.144571 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:34.144388    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:34.145283 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:34.145259    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:16:35.375283 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.375243    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"]
Oct 31 03:16:35.383642 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.383622    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.444178 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.444144    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.444336 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.444185    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.444336 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.444209    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.444488 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.444353    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.444488 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.444462    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.444573 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.444524    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.444573 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.444560    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.444651 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.444582    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8lc\" (UniqueName: \"kubernetes.io/projected/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-kube-api-access-vj8lc\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.444699 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.444686    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.470185 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.470150    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"]
Oct 31 03:16:35.545329 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.545285    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.545329 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.545330    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.545653 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.545366    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.545653 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.545398    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.545653 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.545447    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.545653 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.545521    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.545896 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.545665    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.545896 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.545726    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.545896 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.545760    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8lc\" (UniqueName: \"kubernetes.io/projected/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-kube-api-access-vj8lc\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.545896 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.545853    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.546108 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.545950    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.546108 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.546060    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.546286 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.546267    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.546368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.546351    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.547827 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.547802    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.547968 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.547949    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.599703 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.599668    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.600541 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.600509    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8lc\" (UniqueName: \"kubernetes.io/projected/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-kube-api-access-vj8lc\") pod \"maas-default-gateway-openshift-default-6886b96f9-dffz2\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:35.696584 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:35.696497    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:36.111003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:36.110976    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"]
Oct 31 03:16:36.113537 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:16:36.113511    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf0d481_00ac_4262_8bc7_224f1ff8f54b.slice/crio-7f38c06e8ecb49f050ebfaf9d9f0ab140d68f83833f29b6467943503f6d0167e WatchSource:0}: Error finding container 7f38c06e8ecb49f050ebfaf9d9f0ab140d68f83833f29b6467943503f6d0167e: Status 404 returned error can't find the container with id 7f38c06e8ecb49f050ebfaf9d9f0ab140d68f83833f29b6467943503f6d0167e
Oct 31 03:16:36.115815 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:36.115759    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:16:36.115900 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:36.115863    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:16:36.115935 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:36.115903    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:16:36.306377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:36.306338    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2" event={"ID":"aaf0d481-00ac-4262-8bc7-224f1ff8f54b","Type":"ContainerStarted","Data":"bfdfc0c74e76f95bae2fc4eaa25f2d30eed5a1ee18bdf7d835834bf74c2f1466"}
Oct 31 03:16:36.306551 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:36.306400    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2" event={"ID":"aaf0d481-00ac-4262-8bc7-224f1ff8f54b","Type":"ContainerStarted","Data":"7f38c06e8ecb49f050ebfaf9d9f0ab140d68f83833f29b6467943503f6d0167e"}
Oct 31 03:16:36.417183 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:36.417068    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2" podStartSLOduration=1.417052265 podStartE2EDuration="1.417052265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:36.414928605 +0000 UTC m=+549.381261948" watchObservedRunningTime="2026-10-31 03:16:36.417052265 +0000 UTC m=+549.383385609"
Oct 31 03:16:36.696705 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:36.696616    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:36.984757 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:16:36.984702    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:16:37.698972 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:37.698935    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dffz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 03:16:37.699362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:37.698989    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2" podUID="aaf0d481-00ac-4262-8bc7-224f1ff8f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 03:16:38.697541 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:38.697500    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dffz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 03:16:38.697731 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:38.697574    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2" podUID="aaf0d481-00ac-4262-8bc7-224f1ff8f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 03:16:39.697145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:39.697063    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dffz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 03:16:39.697145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:39.697121    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2" podUID="aaf0d481-00ac-4262-8bc7-224f1ff8f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 03:16:40.563076 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:40.563037    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:16:40.697295 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:40.697250    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dffz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 03:16:40.697690 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:40.697308    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2" podUID="aaf0d481-00ac-4262-8bc7-224f1ff8f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 03:16:41.697200 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:41.697162    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dffz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 03:16:41.697389 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:41.697216    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2" podUID="aaf0d481-00ac-4262-8bc7-224f1ff8f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 03:16:41.854828 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:41.854704    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:16:42.076992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:42.076953    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:16:42.302420 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:42.302379    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:16:42.697338 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:42.697294    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dffz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 31 03:16:42.697548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:42.697349    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2" podUID="aaf0d481-00ac-4262-8bc7-224f1ff8f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 31 03:16:44.405098 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:44.405062    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:44.405574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:44.405503    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:44.406622 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:44.406594    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:16:44.667431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:16:44.667348    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:03.115824 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:03.115767    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:06.983518 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:17:06.983480    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:17:10.959981 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:10.959943    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:11.209398 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:11.209358    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:15.160019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:15.159984    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:15.399321 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:15.399282    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:15.705953 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:15.705888    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:16.001930 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.001895    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:16.197880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.197842    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"]
Oct 31 03:17:16.201258 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.201240    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.253466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.253374    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dsst\" (UniqueName: \"kubernetes.io/projected/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-kube-api-access-6dsst\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.253466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.253421    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-workload-socket\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.253704 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.253500    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-envoy\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.253704 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.253539    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-podinfo\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.253704 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.253558    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-token\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.253704 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.253663    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-data\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.253929 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.253705    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istiod-ca-cert\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.253929 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.253735    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-workload-certs\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.253929 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.253781    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-credential-socket\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.255966 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.255939    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:16.263283 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.263250    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"]
Oct 31 03:17:16.319167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.319135    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2f427162-openshift-default-dockercfg-qtcbr\""
Oct 31 03:17:16.354780 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.354738    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dsst\" (UniqueName: \"kubernetes.io/projected/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-kube-api-access-6dsst\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.354991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.354802    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-workload-socket\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.354991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.354843    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-envoy\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.354991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.354894    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-podinfo\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.354991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.354923    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-token\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.354991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.354961    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-data\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.355249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.354993    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istiod-ca-cert\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.355249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.355028    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-workload-certs\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.355249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.355068    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-credential-socket\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.355249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.355238    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-workload-socket\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.355876 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.355845    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-credential-socket\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.356055 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.356030    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-data\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.357890 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.356253    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-workload-certs\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.358396 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.358321    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istiod-ca-cert\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.370427 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.370148    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-envoy\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.372590 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.372511    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-podinfo\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.410950 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.410912    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-token\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.410950 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.410923    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dsst\" (UniqueName: \"kubernetes.io/projected/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-kube-api-access-6dsst\") pod \"e2e-ait-2f427162-openshift-default-8b8547849-9wg6j\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") " pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.468682 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.468643    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:16.515462 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.515377    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:16.698545 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.698509    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:16.956442 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.956398    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"]
Oct 31 03:17:16.959028 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:17:16.958989    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb621eb_a9f8_4a9e_8881_82d61fe283f6.slice/crio-9e3f34de35d93fb7ce94a2531131fdcad06989a3c62d57eb841a726ca83b2af2 WatchSource:0}: Error finding container 9e3f34de35d93fb7ce94a2531131fdcad06989a3c62d57eb841a726ca83b2af2: Status 404 returned error can't find the container with id 9e3f34de35d93fb7ce94a2531131fdcad06989a3c62d57eb841a726ca83b2af2
Oct 31 03:17:16.961641 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.961599    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:17:16.961854 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.961778    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:17:16.961978 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:16.961960    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:17:17.028578 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:17.028538    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:17.318677 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:17.318638    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:17.489183 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:17.489137    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j" event={"ID":"0cb621eb-a9f8-4a9e-8881-82d61fe283f6","Type":"ContainerStarted","Data":"f38b2c230fcde32d5cbd9c8c8fbe6066d71ddb3049eb0e1da287f41bd8bfd6ca"}
Oct 31 03:17:17.489377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:17.489259    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j" event={"ID":"0cb621eb-a9f8-4a9e-8881-82d61fe283f6","Type":"ContainerStarted","Data":"9e3f34de35d93fb7ce94a2531131fdcad06989a3c62d57eb841a726ca83b2af2"}
Oct 31 03:17:17.516478 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:17.516447    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:17.602531 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:17.602415    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j" podStartSLOduration=1.602397844 podStartE2EDuration="1.602397844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:17.601630466 +0000 UTC m=+590.567963809" watchObservedRunningTime="2026-10-31 03:17:17.602397844 +0000 UTC m=+590.568731190"
Oct 31 03:17:17.876961 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:17.876867    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:18.114881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:18.114846    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:18.398535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:18.398500    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:18.521588 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:18.521554    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:18.620522 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:18.620486    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:19.498373 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:19.498341    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:19.499224 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:19.499206    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:17:19.704292 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:19.704230    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:19.940203 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:19.940118    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:20.346217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:20.346177    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:20.653421 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:20.653341    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:21.454041 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:21.454005    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:21.676177 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:21.676142    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:22.353059 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:22.353025    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:22.648412 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:22.648331    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:22.881821 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:22.881777    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:23.079877 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.079840    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"]
Oct 31 03:17:23.083252 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.083232    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.137141 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.137112    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-15000fe0-openshift-default-dockercfg-fjzpx\""
Oct 31 03:17:23.185311 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.185279    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"]
Oct 31 03:17:23.227683 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.227633    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-envoy\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.230020 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.227765    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-credential-socket\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.230020 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.227885    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-podinfo\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.230020 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.227947    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-data\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.230020 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.228006    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcw6j\" (UniqueName: \"kubernetes.io/projected/195f8e1b-34bf-41ff-a708-17364e8a42fd-kube-api-access-tcw6j\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.230020 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.228086    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-token\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.230020 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.228231    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-workload-socket\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.230020 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.228706    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-workload-certs\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.230020 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.228757    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195f8e1b-34bf-41ff-a708-17364e8a42fd-istiod-ca-cert\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.261186 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.261151    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:23.330262 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.330166    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-envoy\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.330262 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.330224    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-credential-socket\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.330499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.330270    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-podinfo\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.330499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.330297    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-data\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.330499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.330321    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcw6j\" (UniqueName: \"kubernetes.io/projected/195f8e1b-34bf-41ff-a708-17364e8a42fd-kube-api-access-tcw6j\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.330499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.330377    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-token\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.330499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.330430    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-workload-socket\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.330499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.330479    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-workload-certs\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.330848 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.330505    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195f8e1b-34bf-41ff-a708-17364e8a42fd-istiod-ca-cert\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.330848 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.330731    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-credential-socket\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.331029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.331004    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-data\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.331089 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.331062    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-workload-certs\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.331155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.331131    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-workload-socket\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.331383 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.331363    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195f8e1b-34bf-41ff-a708-17364e8a42fd-istiod-ca-cert\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.332688 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.332663    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-envoy\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.332943 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.332927    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-podinfo\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.384192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.384155    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcw6j\" (UniqueName: \"kubernetes.io/projected/195f8e1b-34bf-41ff-a708-17364e8a42fd-kube-api-access-tcw6j\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.384353 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.384335    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-token\") pod \"e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") " pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.397298 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.397271    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:23.514846 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.514812    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:23.814579 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.814555    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"]
Oct 31 03:17:23.816276 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:17:23.816238    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195f8e1b_34bf_41ff_a708_17364e8a42fd.slice/crio-38b5602eeff28a6418ad2538d2222dedde04c549b1ac25af1ce64951b28915b5 WatchSource:0}: Error finding container 38b5602eeff28a6418ad2538d2222dedde04c549b1ac25af1ce64951b28915b5: Status 404 returned error can't find the container with id 38b5602eeff28a6418ad2538d2222dedde04c549b1ac25af1ce64951b28915b5
Oct 31 03:17:23.818480 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.818446    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:17:23.818589 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.818511    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:17:23.818589 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.818540    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:17:23.891251 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:23.891219    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:24.198244 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.198113    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:24.219605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.219573    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"]
Oct 31 03:17:24.223069 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.223049    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"
Oct 31 03:17:24.274168 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.274135    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-bac33a89\""
Oct 31 03:17:24.274321 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.274188    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-bac33a89-dockercfg-bp7fn\""
Oct 31 03:17:24.278957 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.278928    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"]
Oct 31 03:17:24.342350 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.342308    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a019c6-1460-4dc1-b322-1a8e7afa4aed-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg\" (UID: \"76a019c6-1460-4dc1-b322-1a8e7afa4aed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"
Oct 31 03:17:24.342549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.342392    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4nf4\" (UniqueName: \"kubernetes.io/projected/76a019c6-1460-4dc1-b322-1a8e7afa4aed-kube-api-access-s4nf4\") pod \"payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg\" (UID: \"76a019c6-1460-4dc1-b322-1a8e7afa4aed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"
Oct 31 03:17:24.372779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.372742    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"]
Oct 31 03:17:24.376359 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.376342    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"
Oct 31 03:17:24.435514 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.435483    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"]
Oct 31 03:17:24.443383 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.443355    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a019c6-1460-4dc1-b322-1a8e7afa4aed-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg\" (UID: \"76a019c6-1460-4dc1-b322-1a8e7afa4aed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"
Oct 31 03:17:24.443539 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.443404    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4nf4\" (UniqueName: \"kubernetes.io/projected/76a019c6-1460-4dc1-b322-1a8e7afa4aed-kube-api-access-s4nf4\") pod \"payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg\" (UID: \"76a019c6-1460-4dc1-b322-1a8e7afa4aed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"
Oct 31 03:17:24.443539 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.443455    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f82m\" (UniqueName: \"kubernetes.io/projected/6f60debc-d81d-44ab-807e-c6e3418a6db8-kube-api-access-6f82m\") pod \"payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm\" (UID: \"6f60debc-d81d-44ab-807e-c6e3418a6db8\") " pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"
Oct 31 03:17:24.443539 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.443474    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f60debc-d81d-44ab-807e-c6e3418a6db8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm\" (UID: \"6f60debc-d81d-44ab-807e-c6e3418a6db8\") " pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"
Oct 31 03:17:24.444122 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.444102    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a019c6-1460-4dc1-b322-1a8e7afa4aed-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg\" (UID: \"76a019c6-1460-4dc1-b322-1a8e7afa4aed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"
Oct 31 03:17:24.497780 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.497749    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4nf4\" (UniqueName: \"kubernetes.io/projected/76a019c6-1460-4dc1-b322-1a8e7afa4aed-kube-api-access-s4nf4\") pod \"payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg\" (UID: \"76a019c6-1460-4dc1-b322-1a8e7afa4aed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"
Oct 31 03:17:24.523099 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.523067    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb" event={"ID":"195f8e1b-34bf-41ff-a708-17364e8a42fd","Type":"ContainerStarted","Data":"05ad7cde6fed2c5b32485c1f6f5255cd2173a788bbae64d208a4c279a381d00d"}
Oct 31 03:17:24.523277 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.523130    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb" event={"ID":"195f8e1b-34bf-41ff-a708-17364e8a42fd","Type":"ContainerStarted","Data":"38b5602eeff28a6418ad2538d2222dedde04c549b1ac25af1ce64951b28915b5"}
Oct 31 03:17:24.535231 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.535208    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"
Oct 31 03:17:24.544580 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.544552    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f82m\" (UniqueName: \"kubernetes.io/projected/6f60debc-d81d-44ab-807e-c6e3418a6db8-kube-api-access-6f82m\") pod \"payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm\" (UID: \"6f60debc-d81d-44ab-807e-c6e3418a6db8\") " pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"
Oct 31 03:17:24.544684 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.544594    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f60debc-d81d-44ab-807e-c6e3418a6db8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm\" (UID: \"6f60debc-d81d-44ab-807e-c6e3418a6db8\") " pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"
Oct 31 03:17:24.545337 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.545314    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f60debc-d81d-44ab-807e-c6e3418a6db8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm\" (UID: \"6f60debc-d81d-44ab-807e-c6e3418a6db8\") " pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"
Oct 31 03:17:24.601363 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.601333    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f82m\" (UniqueName: \"kubernetes.io/projected/6f60debc-d81d-44ab-807e-c6e3418a6db8-kube-api-access-6f82m\") pod \"payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm\" (UID: \"6f60debc-d81d-44ab-807e-c6e3418a6db8\") " pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"
Oct 31 03:17:24.642230 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.642167    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb" podStartSLOduration=1.6421524170000001 podStartE2EDuration="1.642152417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:24.640507881 +0000 UTC m=+597.606841237" watchObservedRunningTime="2026-10-31 03:17:24.642152417 +0000 UTC m=+597.608485761"
Oct 31 03:17:24.691159 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.691129    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"
Oct 31 03:17:24.906180 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:24.906091    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"]
Oct 31 03:17:24.909003 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:17:24.908974    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a019c6_1460_4dc1_b322_1a8e7afa4aed.slice/crio-3625c1c245f7cea17435c82ca280e5c29f8c65139161c4b570756e99bd87fadd WatchSource:0}: Error finding container 3625c1c245f7cea17435c82ca280e5c29f8c65139161c4b570756e99bd87fadd: Status 404 returned error can't find the container with id 3625c1c245f7cea17435c82ca280e5c29f8c65139161c4b570756e99bd87fadd
Oct 31 03:17:25.054902 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:25.054872    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"]
Oct 31 03:17:25.057665 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:17:25.057642    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f60debc_d81d_44ab_807e_c6e3418a6db8.slice/crio-b9e200bb41e2df40088cc4b32478e57ebf7910a16df0e8904f8643553f10dd67 WatchSource:0}: Error finding container b9e200bb41e2df40088cc4b32478e57ebf7910a16df0e8904f8643553f10dd67: Status 404 returned error can't find the container with id b9e200bb41e2df40088cc4b32478e57ebf7910a16df0e8904f8643553f10dd67
Oct 31 03:17:25.397867 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:25.397828    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:25.403085 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:25.403055    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:25.529546 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:25.529500    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm" event={"ID":"6f60debc-d81d-44ab-807e-c6e3418a6db8","Type":"ContainerStarted","Data":"b84e791012cb4a613860aa9732855ff733f10f51f303f60038cc7ab9af5428f5"}
Oct 31 03:17:25.529751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:25.529565    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm" event={"ID":"6f60debc-d81d-44ab-807e-c6e3418a6db8","Type":"ContainerStarted","Data":"b9e200bb41e2df40088cc4b32478e57ebf7910a16df0e8904f8643553f10dd67"}
Oct 31 03:17:25.529751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:25.529597    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"
Oct 31 03:17:25.531079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:25.531053    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg" event={"ID":"76a019c6-1460-4dc1-b322-1a8e7afa4aed","Type":"ContainerStarted","Data":"c4a45d505466fb0959ccfcb115266e429de46a4e657bb848b8229c4566cc654f"}
Oct 31 03:17:25.531157 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:25.531099    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg" event={"ID":"76a019c6-1460-4dc1-b322-1a8e7afa4aed","Type":"ContainerStarted","Data":"3625c1c245f7cea17435c82ca280e5c29f8c65139161c4b570756e99bd87fadd"}
Oct 31 03:17:25.531194 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:25.531150    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"
Oct 31 03:17:25.531411 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:25.531395    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:25.532327 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:25.532310    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:17:25.768505 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:25.768441    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg" podStartSLOduration=1.7684209960000001 podStartE2EDuration="1.768420996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:25.767642444 +0000 UTC m=+598.733975788" watchObservedRunningTime="2026-10-31 03:17:25.768420996 +0000 UTC m=+598.734754341"
Oct 31 03:17:25.875059 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:25.874768    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm" podStartSLOduration=1.874748459 podStartE2EDuration="1.874748459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:25.874065779 +0000 UTC m=+598.840399122" watchObservedRunningTime="2026-10-31 03:17:25.874748459 +0000 UTC m=+598.841081804"
Oct 31 03:17:25.958164 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:25.958133    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:26.199062 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.198984    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:26.256817 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.256760    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"]
Oct 31 03:17:26.260668 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.260644    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:26.312162 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.312128    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4072c5d1-serving-cert\""
Oct 31 03:17:26.314920 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.314891    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"]
Oct 31 03:17:26.362913 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.362875    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8c75\" (UniqueName: \"kubernetes.io/projected/f3051a5f-94f1-4a26-a927-d8bd71d1590a-kube-api-access-p8c75\") pod \"maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:26.363090 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.362928    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-metrics-certs\") pod \"maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:26.363090 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.362987    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3051a5f-94f1-4a26-a927-d8bd71d1590a-openshift-service-ca\") pod \"maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:26.363217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.363111    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-maas-api-tls\") pod \"maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:26.464714 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.464628    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-metrics-certs\") pod \"maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:26.464714 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.464667    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3051a5f-94f1-4a26-a927-d8bd71d1590a-openshift-service-ca\") pod \"maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:26.464975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.464769    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-maas-api-tls\") pod \"maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:26.464975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.464889    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8c75\" (UniqueName: \"kubernetes.io/projected/f3051a5f-94f1-4a26-a927-d8bd71d1590a-kube-api-access-p8c75\") pod \"maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:26.465084 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:17:26.464979    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-serving-cert: secret "maas-api-e2e-ait-4072c5d1-serving-cert" not found
Oct 31 03:17:26.465084 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:17:26.465053    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-maas-api-tls podName:f3051a5f-94f1-4a26-a927-d8bd71d1590a nodeName:}" failed. No retries permitted until 2026-10-31 03:17:26.965031242 +0000 UTC m=+599.931364577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-maas-api-tls") pod "maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l" (UID: "f3051a5f-94f1-4a26-a927-d8bd71d1590a") : secret "maas-api-e2e-ait-4072c5d1-serving-cert" not found
Oct 31 03:17:26.465499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.465477    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3051a5f-94f1-4a26-a927-d8bd71d1590a-openshift-service-ca\") pod \"maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:26.467067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.467039    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-metrics-certs\") pod \"maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:26.526668 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.526633    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8c75\" (UniqueName: \"kubernetes.io/projected/f3051a5f-94f1-4a26-a927-d8bd71d1590a-kube-api-access-p8c75\") pod \"maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:26.613148 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.613115    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:26.833846 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.833814    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:26.970257 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.970225    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-maas-api-tls\") pod \"maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:26.972648 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:26.972625    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-maas-api-tls\") pod \"maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:27.179575 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:27.179476    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:27.193894 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:27.193868    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:17:27.196043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:27.196017    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:17:27.554551 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:27.554524    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"]
Oct 31 03:17:27.557197 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:17:27.557172    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3051a5f_94f1_4a26_a927_d8bd71d1590a.slice/crio-90781c5968b096fbfc7fe229154956b9f74499cd3dd821f0ab10d361891b4d23 WatchSource:0}: Error finding container 90781c5968b096fbfc7fe229154956b9f74499cd3dd821f0ab10d361891b4d23: Status 404 returned error can't find the container with id 90781c5968b096fbfc7fe229154956b9f74499cd3dd821f0ab10d361891b4d23
Oct 31 03:17:28.547277 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:28.547242    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l" event={"ID":"f3051a5f-94f1-4a26-a927-d8bd71d1590a","Type":"ContainerStarted","Data":"90781c5968b096fbfc7fe229154956b9f74499cd3dd821f0ab10d361891b4d23"}
Oct 31 03:17:28.594818 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:28.594769    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:28.804365 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:28.804263    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:29.555038 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:29.554998    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l" event={"ID":"f3051a5f-94f1-4a26-a927-d8bd71d1590a","Type":"ContainerStarted","Data":"2595af3a0903c6b66ed8925d73cb8b3fab9b8b9ab9e9fe715ff1c5377bf51a1f"}
Oct 31 03:17:29.555548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:29.555105    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:29.665608 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:29.665552    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l" podStartSLOduration=2.586455196 podStartE2EDuration="3.66553703s" totalImagesPullingTime="1.079081834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:29.663664504 +0000 UTC m=+602.629997848" watchObservedRunningTime="2026-10-31 03:17:29.66553703 +0000 UTC m=+602.631870377"
Oct 31 03:17:30.934648 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:30.934611    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:31.214876 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:31.214775    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:31.449880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:31.449847    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:31.866295 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:31.866258    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"]
Oct 31 03:17:31.870034 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:31.870016    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:31.969078 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:31.969040    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-bac33a89-serving-cert\""
Oct 31 03:17:31.971050 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:31.971026    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"]
Oct 31 03:17:32.022406 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.022367    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8856\" (UniqueName: \"kubernetes.io/projected/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-kube-api-access-j8856\") pod \"maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:32.022605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.022521    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-openshift-service-ca\") pod \"maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:32.022605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.022551    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-metrics-certs\") pod \"maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:32.022605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.022578    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-maas-api-tls\") pod \"maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:32.124060 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.123969    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-openshift-service-ca\") pod \"maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:32.124060 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.124007    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-metrics-certs\") pod \"maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:32.124060 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.124039    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-maas-api-tls\") pod \"maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:32.124341 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.124136    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8856\" (UniqueName: \"kubernetes.io/projected/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-kube-api-access-j8856\") pod \"maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:32.124341 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:17:32.124209    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-serving-cert: secret "maas-api-e2e-migrate-bac33a89-serving-cert" not found
Oct 31 03:17:32.124341 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:17:32.124291    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-maas-api-tls podName:73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f nodeName:}" failed. No retries permitted until 2026-10-31 03:17:32.624270395 +0000 UTC m=+605.590603736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-maas-api-tls") pod "maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67" (UID: "73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f") : secret "maas-api-e2e-migrate-bac33a89-serving-cert" not found
Oct 31 03:17:32.124869 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.124846    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-openshift-service-ca\") pod \"maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:32.126897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.126874    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-metrics-certs\") pod \"maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:32.165192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.165158    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:32.177876 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.177848    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8856\" (UniqueName: \"kubernetes.io/projected/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-kube-api-access-j8856\") pod \"maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:32.383783 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.383691    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:32.594738 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.594702    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:32.630205 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.630173    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-maas-api-tls\") pod \"maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:32.632561 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.632535    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-maas-api-tls\") pod \"maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:32.782466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:32.782432    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:33.149626 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:17:33.149594    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fa9c5e_fe56_4f7f_8a57_e7a5dadda41f.slice/crio-7954825ee486a105d9917b44983e7f2e71b462aa316b086106fa4992304251e4 WatchSource:0}: Error finding container 7954825ee486a105d9917b44983e7f2e71b462aa316b086106fa4992304251e4: Status 404 returned error can't find the container with id 7954825ee486a105d9917b44983e7f2e71b462aa316b086106fa4992304251e4
Oct 31 03:17:33.152074 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:33.152054    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"]
Oct 31 03:17:33.368136 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:33.368096    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:33.583679 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:33.583637    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67" event={"ID":"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f","Type":"ContainerStarted","Data":"7954825ee486a105d9917b44983e7f2e71b462aa316b086106fa4992304251e4"}
Oct 31 03:17:33.601134 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:33.601105    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:34.590335 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:34.590301    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67" event={"ID":"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f","Type":"ContainerStarted","Data":"c4a8b295869a8c1fba15549a3068bbeb16bd119cccdae16a0a0563e87c734588"}
Oct 31 03:17:34.590738 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:34.590414    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:34.698962 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:34.698882    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67" podStartSLOduration=2.843461365 podStartE2EDuration="3.698861627s" totalImagesPullingTime="855.400262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:34.698424294 +0000 UTC m=+607.664757639" watchObservedRunningTime="2026-10-31 03:17:34.698861627 +0000 UTC m=+607.665195046"
Oct 31 03:17:35.565828 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:35.565775    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:17:36.540116 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:36.540086    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"
Oct 31 03:17:36.540558 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:36.540165    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"
Oct 31 03:17:36.984163 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:17:36.984132    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:17:37.399134 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:37.399017    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:40.601428 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:40.601399    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:17:42.255142 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.255109    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:42.310519 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.310490    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"]
Oct 31 03:17:42.313826 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.313808    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:42.331971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.331944    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpb72\" (UniqueName: \"kubernetes.io/projected/7d4a653b-999b-4878-aaae-c29887c4bcab-kube-api-access-rpb72\") pod \"maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:42.332103 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.331998    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-metrics-certs\") pod \"maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:42.332157 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.332107    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-maas-api-tls\") pod \"maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:42.332157 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.332137    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4a653b-999b-4878-aaae-c29887c4bcab-openshift-service-ca\") pod \"maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:42.367803 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.367759    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2f427162-serving-cert\""
Oct 31 03:17:42.414602 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.414566    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"]
Oct 31 03:17:42.432750 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.432717    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-maas-api-tls\") pod \"maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:42.432963 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.432767    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4a653b-999b-4878-aaae-c29887c4bcab-openshift-service-ca\") pod \"maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:42.432963 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.432863    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpb72\" (UniqueName: \"kubernetes.io/projected/7d4a653b-999b-4878-aaae-c29887c4bcab-kube-api-access-rpb72\") pod \"maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:42.432963 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:17:42.432912    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-serving-cert: secret "maas-api-e2e-ait-2f427162-serving-cert" not found
Oct 31 03:17:42.433112 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.432918    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-metrics-certs\") pod \"maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:42.433112 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:17:42.432983    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-maas-api-tls podName:7d4a653b-999b-4878-aaae-c29887c4bcab nodeName:}" failed. No retries permitted until 2026-10-31 03:17:42.932962756 +0000 UTC m=+615.899296081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-maas-api-tls") pod "maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r" (UID: "7d4a653b-999b-4878-aaae-c29887c4bcab") : secret "maas-api-e2e-ait-2f427162-serving-cert" not found
Oct 31 03:17:42.433456 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.433439    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4a653b-999b-4878-aaae-c29887c4bcab-openshift-service-ca\") pod \"maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:42.435493 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.435469    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-metrics-certs\") pod \"maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:42.486427 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.486374    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpb72\" (UniqueName: \"kubernetes.io/projected/7d4a653b-999b-4878-aaae-c29887c4bcab-kube-api-access-rpb72\") pod \"maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:42.504640 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.504612    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:42.724297 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.724211    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:42.933230 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.933198    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:42.938509 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.938484    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-maas-api-tls\") pod \"maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:42.940886 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:42.940860    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-maas-api-tls\") pod \"maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:43.226295 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:43.226254    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:43.597239 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:43.597213    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"]
Oct 31 03:17:43.599706 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:17:43.599675    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4a653b_999b_4878_aaae_c29887c4bcab.slice/crio-76e00853bff8c68dae5870f362dfc89b42ffa84afdb6ffa90fd15e5d83871134 WatchSource:0}: Error finding container 76e00853bff8c68dae5870f362dfc89b42ffa84afdb6ffa90fd15e5d83871134: Status 404 returned error can't find the container with id 76e00853bff8c68dae5870f362dfc89b42ffa84afdb6ffa90fd15e5d83871134
Oct 31 03:17:43.634131 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:43.634098    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r" event={"ID":"7d4a653b-999b-4878-aaae-c29887c4bcab","Type":"ContainerStarted","Data":"76e00853bff8c68dae5870f362dfc89b42ffa84afdb6ffa90fd15e5d83871134"}
Oct 31 03:17:44.640190 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:44.640153    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r" event={"ID":"7d4a653b-999b-4878-aaae-c29887c4bcab","Type":"ContainerStarted","Data":"b25f5fddd45496ea102569d347c8e27270dc7a823ef5e3b084780d7e6751abbe"}
Oct 31 03:17:44.640580 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:44.640220    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:44.787918 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:44.787809    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r" podStartSLOduration=1.875403822 podStartE2EDuration="2.787781308s" totalImagesPullingTime="912.377486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:44.786267822 +0000 UTC m=+617.752601166" watchObservedRunningTime="2026-10-31 03:17:44.787781308 +0000 UTC m=+617.754114653"
Oct 31 03:17:45.003027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:45.002991    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:47.445864 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.445831    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"]
Oct 31 03:17:47.449608 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.449592    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:47.484233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.484207    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-metrics-certs\") pod \"maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:47.484367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.484258    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-maas-api-tls\") pod \"maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:47.484367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.484289    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a24e69-35fb-45c5-a85e-14d410aa89dc-openshift-service-ca\") pod \"maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:47.484367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.484324    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjln2\" (UniqueName: \"kubernetes.io/projected/b9a24e69-35fb-45c5-a85e-14d410aa89dc-kube-api-access-kjln2\") pod \"maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:47.547170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.547142    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-15000fe0-serving-cert\""
Oct 31 03:17:47.549199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.549175    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"]
Oct 31 03:17:47.585315 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.585283    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjln2\" (UniqueName: \"kubernetes.io/projected/b9a24e69-35fb-45c5-a85e-14d410aa89dc-kube-api-access-kjln2\") pod \"maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:47.585483 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.585400    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-metrics-certs\") pod \"maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:47.585483 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.585452    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-maas-api-tls\") pod \"maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:47.585483 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.585469    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a24e69-35fb-45c5-a85e-14d410aa89dc-openshift-service-ca\") pod \"maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:47.585703 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:17:47.585570    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-serving-cert: secret "maas-api-e2e-derive-15000fe0-serving-cert" not found
Oct 31 03:17:47.585703 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:17:47.585649    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-maas-api-tls podName:b9a24e69-35fb-45c5-a85e-14d410aa89dc nodeName:}" failed. No retries permitted until 2026-10-31 03:17:48.085629222 +0000 UTC m=+621.051962548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-maas-api-tls") pod "maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj" (UID: "b9a24e69-35fb-45c5-a85e-14d410aa89dc") : secret "maas-api-e2e-derive-15000fe0-serving-cert" not found
Oct 31 03:17:47.586237 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.586211    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a24e69-35fb-45c5-a85e-14d410aa89dc-openshift-service-ca\") pod \"maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:47.588117 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.588093    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-metrics-certs\") pod \"maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:47.639043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.639019    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjln2\" (UniqueName: \"kubernetes.io/projected/b9a24e69-35fb-45c5-a85e-14d410aa89dc-kube-api-access-kjln2\") pod \"maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:47.731307 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.731178    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:47.946652 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:47.946614    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:48.091711 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:48.091669    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-maas-api-tls\") pod \"maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:48.094036 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:48.094013    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-maas-api-tls\") pod \"maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:48.154120 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:48.154086    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:48.362118 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:48.362037    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:48.728984 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:48.728959    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"]
Oct 31 03:17:48.731645 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:17:48.731615    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a24e69_35fb_45c5_a85e_14d410aa89dc.slice/crio-6f0e1f823618c6407eb6abce633cc17eb5a1fbc633c0addc931ac3a5e9fd9c1c WatchSource:0}: Error finding container 6f0e1f823618c6407eb6abce633cc17eb5a1fbc633c0addc931ac3a5e9fd9c1c: Status 404 returned error can't find the container with id 6f0e1f823618c6407eb6abce633cc17eb5a1fbc633c0addc931ac3a5e9fd9c1c
Oct 31 03:17:49.666218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:49.666182    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj" event={"ID":"b9a24e69-35fb-45c5-a85e-14d410aa89dc","Type":"ContainerStarted","Data":"6f0e1f823618c6407eb6abce633cc17eb5a1fbc633c0addc931ac3a5e9fd9c1c"}
Oct 31 03:17:50.663119 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:50.663091    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:17:50.678573 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:50.678536    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj" event={"ID":"b9a24e69-35fb-45c5-a85e-14d410aa89dc","Type":"ContainerStarted","Data":"d27d26037dc5ba5dcea843031233446e398cf0d205059571de7ab818090924f3"}
Oct 31 03:17:50.678712 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:50.678629    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:50.879665 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:50.879605    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj" podStartSLOduration=2.91789565 podStartE2EDuration="3.879584689s" totalImagesPullingTime="961.689039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:17:50.879003797 +0000 UTC m=+623.845337143" watchObservedRunningTime="2026-10-31 03:17:50.879584689 +0000 UTC m=+623.845918036"
Oct 31 03:17:53.120115 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:53.120075    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:53.324480 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:53.324444    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:53.531737 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:53.531700    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:56.690234 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:56.690203    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:17:58.292426 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:58.292391    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:17:58.717116 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:58.717082    4900 generic.go:356] "Generic (PLEG): container finished" podID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" containerID="f38b2c230fcde32d5cbd9c8c8fbe6066d71ddb3049eb0e1da287f41bd8bfd6ca" exitCode=0
Oct 31 03:17:58.717284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:58.717157    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j" event={"ID":"0cb621eb-a9f8-4a9e-8881-82d61fe283f6","Type":"ContainerDied","Data":"f38b2c230fcde32d5cbd9c8c8fbe6066d71ddb3049eb0e1da287f41bd8bfd6ca"}
Oct 31 03:17:58.717576 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:58.717557    4900 scope.go:122] "RemoveContainer" containerID="f38b2c230fcde32d5cbd9c8c8fbe6066d71ddb3049eb0e1da287f41bd8bfd6ca"
Oct 31 03:17:58.718912 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:58.718885    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:17:58.718981 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:58.718953    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:17:58.719018 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:58.718985    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:17:59.724108 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:17:59.724073    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j" event={"ID":"0cb621eb-a9f8-4a9e-8881-82d61fe283f6","Type":"ContainerStarted","Data":"bd98d8a280b09865c0962225f7009c47bef57b5ed987e4819bbd290606d7f65f"}
Oct 31 03:18:00.515862 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:00.515829    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:18:00.517178 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:00.517138    4900 patch_prober.go:29] interesting pod/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 03:18:00.517244 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:00.517210    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j" podUID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 03:18:01.516218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:01.516182    4900 patch_prober.go:29] interesting pod/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 03:18:01.516619 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:01.516249    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j" podUID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 03:18:02.516090 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:02.516051    4900 patch_prober.go:29] interesting pod/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 03:18:02.516283 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:02.516117    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j" podUID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 03:18:02.711263 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:02.711224    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:03.305385 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:03.305337    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:03.516177 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:03.516137    4900 patch_prober.go:29] interesting pod/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 03:18:03.516339 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:03.516187    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j" podUID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 03:18:04.498487 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:04.498432    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:18:04.516327 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:04.516275    4900 patch_prober.go:29] interesting pod/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Oct 31 03:18:04.516850 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:04.516354    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j" podUID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Oct 31 03:18:05.521447 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:05.521410    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:18:05.522808 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:05.522762    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:18:06.984778 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:06.984743    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:18:08.497033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:08.496996    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:08.819825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:08.819771    4900 generic.go:356] "Generic (PLEG): container finished" podID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerID="05ad7cde6fed2c5b32485c1f6f5255cd2173a788bbae64d208a4c279a381d00d" exitCode=0
Oct 31 03:18:08.820003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:08.819831    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb" event={"ID":"195f8e1b-34bf-41ff-a708-17364e8a42fd","Type":"ContainerDied","Data":"05ad7cde6fed2c5b32485c1f6f5255cd2173a788bbae64d208a4c279a381d00d"}
Oct 31 03:18:08.820212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:08.820198    4900 scope.go:122] "RemoveContainer" containerID="05ad7cde6fed2c5b32485c1f6f5255cd2173a788bbae64d208a4c279a381d00d"
Oct 31 03:18:08.821666 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:08.821640    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:18:08.821731 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:08.821706    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:18:08.821771 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:08.821736    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:18:09.827145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:09.827111    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb" event={"ID":"195f8e1b-34bf-41ff-a708-17364e8a42fd","Type":"ContainerStarted","Data":"1dd83886aa3dee176f719060f34d8140a6d2aad40f223c770e127bceda7cfe17"}
Oct 31 03:18:10.398050 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:10.398011    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:18:10.398984 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:10.398957    4900 patch_prober.go:29] interesting pod/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 03:18:10.399097 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:10.399006    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb" podUID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 03:18:10.531545 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:10.531517    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:18:11.398446 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:11.398414    4900 patch_prober.go:29] interesting pod/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 03:18:11.398879 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:11.398463    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb" podUID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 03:18:12.397696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:12.397659    4900 patch_prober.go:29] interesting pod/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 03:18:12.397901 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:12.397712    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb" podUID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 03:18:13.398102 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:13.398067    4900 patch_prober.go:29] interesting pod/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 03:18:13.398500 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:13.398132    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb" podUID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 03:18:14.140688 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:14.140653    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:14.398252 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:14.398160    4900 patch_prober.go:29] interesting pod/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Oct 31 03:18:14.398252 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:14.398225    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb" podUID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Oct 31 03:18:15.402281 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:15.402243    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:18:15.403246 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:15.403226    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:18:16.184942 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:16.184908    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:18.630624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:18.630592    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:18.902019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:18.901915    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:19.605169 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:19.605138    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:22.158329 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:22.158298    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"]
Oct 31 03:18:22.267625 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:22.267590    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"]
Oct 31 03:18:22.603274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:22.603236    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:22.814320 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:22.814272    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:24.930377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:24.930338    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:27.159154 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.159103    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm" podUID="6f60debc-d81d-44ab-807e-c6e3418a6db8" containerName="payload-processing" containerID="cri-o://b84e791012cb4a613860aa9732855ff733f10f51f303f60038cc7ab9af5428f5" gracePeriod=25
Oct 31 03:18:27.268929 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.268887    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg" podUID="76a019c6-1460-4dc1-b322-1a8e7afa4aed" containerName="payload-pre-processing" containerID="cri-o://c4a45d505466fb0959ccfcb115266e429de46a4e657bb848b8229c4566cc654f" gracePeriod=25
Oct 31 03:18:27.401016 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.400978    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"
Oct 31 03:18:27.518025 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.517999    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"
Oct 31 03:18:27.561148 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.561123    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f60debc-d81d-44ab-807e-c6e3418a6db8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f60debc-d81d-44ab-807e-c6e3418a6db8-plugins-config-volume\") pod \"6f60debc-d81d-44ab-807e-c6e3418a6db8\" (UID: \"6f60debc-d81d-44ab-807e-c6e3418a6db8\") "
Oct 31 03:18:27.561284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.561270    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f60debc-d81d-44ab-807e-c6e3418a6db8-kube-api-access-6f82m\" (UniqueName: \"kubernetes.io/projected/6f60debc-d81d-44ab-807e-c6e3418a6db8-kube-api-access-6f82m\") pod \"6f60debc-d81d-44ab-807e-c6e3418a6db8\" (UID: \"6f60debc-d81d-44ab-807e-c6e3418a6db8\") "
Oct 31 03:18:27.561474 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.561449    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f60debc-d81d-44ab-807e-c6e3418a6db8-plugins-config-volume" pod "6f60debc-d81d-44ab-807e-c6e3418a6db8" (UID: "6f60debc-d81d-44ab-807e-c6e3418a6db8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:27.561553 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.561540    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f60debc-d81d-44ab-807e-c6e3418a6db8-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:27.563171 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.563151    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f60debc-d81d-44ab-807e-c6e3418a6db8-kube-api-access-6f82m" pod "6f60debc-d81d-44ab-807e-c6e3418a6db8" (UID: "6f60debc-d81d-44ab-807e-c6e3418a6db8"). InnerVolumeSpecName "kube-api-access-6f82m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:27.662591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.662559    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76a019c6-1460-4dc1-b322-1a8e7afa4aed-kube-api-access-s4nf4\" (UniqueName: \"kubernetes.io/projected/76a019c6-1460-4dc1-b322-1a8e7afa4aed-kube-api-access-s4nf4\") pod \"76a019c6-1460-4dc1-b322-1a8e7afa4aed\" (UID: \"76a019c6-1460-4dc1-b322-1a8e7afa4aed\") "
Oct 31 03:18:27.662764 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.662604    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76a019c6-1460-4dc1-b322-1a8e7afa4aed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a019c6-1460-4dc1-b322-1a8e7afa4aed-plugins-config-volume\") pod \"76a019c6-1460-4dc1-b322-1a8e7afa4aed\" (UID: \"76a019c6-1460-4dc1-b322-1a8e7afa4aed\") "
Oct 31 03:18:27.662884 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.662874    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f82m\" (UniqueName: \"kubernetes.io/projected/6f60debc-d81d-44ab-807e-c6e3418a6db8-kube-api-access-6f82m\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:27.663009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.662985    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76a019c6-1460-4dc1-b322-1a8e7afa4aed-plugins-config-volume" pod "76a019c6-1460-4dc1-b322-1a8e7afa4aed" (UID: "76a019c6-1460-4dc1-b322-1a8e7afa4aed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:27.664535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.664515    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76a019c6-1460-4dc1-b322-1a8e7afa4aed-kube-api-access-s4nf4" pod "76a019c6-1460-4dc1-b322-1a8e7afa4aed" (UID: "76a019c6-1460-4dc1-b322-1a8e7afa4aed"). InnerVolumeSpecName "kube-api-access-s4nf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:27.764363 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.764324    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4nf4\" (UniqueName: \"kubernetes.io/projected/76a019c6-1460-4dc1-b322-1a8e7afa4aed-kube-api-access-s4nf4\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:27.764363 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.764357    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76a019c6-1460-4dc1-b322-1a8e7afa4aed-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:27.944718 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.944682    4900 generic.go:356] "Generic (PLEG): container finished" podID="6f60debc-d81d-44ab-807e-c6e3418a6db8" containerID="b84e791012cb4a613860aa9732855ff733f10f51f303f60038cc7ab9af5428f5" exitCode=0
Oct 31 03:18:27.944909 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.944726    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"
Oct 31 03:18:27.944909 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.944766    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm" event={"ID":"6f60debc-d81d-44ab-807e-c6e3418a6db8","Type":"ContainerDied","Data":"b84e791012cb4a613860aa9732855ff733f10f51f303f60038cc7ab9af5428f5"}
Oct 31 03:18:27.944909 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.944850    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm" event={"ID":"6f60debc-d81d-44ab-807e-c6e3418a6db8","Type":"ContainerDied","Data":"b9e200bb41e2df40088cc4b32478e57ebf7910a16df0e8904f8643553f10dd67"}
Oct 31 03:18:27.944909 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.944879    4900 scope.go:122] "RemoveContainer" containerID="b84e791012cb4a613860aa9732855ff733f10f51f303f60038cc7ab9af5428f5"
Oct 31 03:18:27.946223 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.946184    4900 generic.go:356] "Generic (PLEG): container finished" podID="76a019c6-1460-4dc1-b322-1a8e7afa4aed" containerID="c4a45d505466fb0959ccfcb115266e429de46a4e657bb848b8229c4566cc654f" exitCode=0
Oct 31 03:18:27.946340 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.946226    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg" event={"ID":"76a019c6-1460-4dc1-b322-1a8e7afa4aed","Type":"ContainerDied","Data":"c4a45d505466fb0959ccfcb115266e429de46a4e657bb848b8229c4566cc654f"}
Oct 31 03:18:27.946340 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.946285    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg" event={"ID":"76a019c6-1460-4dc1-b322-1a8e7afa4aed","Type":"ContainerDied","Data":"3625c1c245f7cea17435c82ca280e5c29f8c65139161c4b570756e99bd87fadd"}
Oct 31 03:18:27.946340 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.946232    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"
Oct 31 03:18:27.953561 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.953538    4900 scope.go:122] "RemoveContainer" containerID="b84e791012cb4a613860aa9732855ff733f10f51f303f60038cc7ab9af5428f5"
Oct 31 03:18:27.953824 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:27.953801    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b84e791012cb4a613860aa9732855ff733f10f51f303f60038cc7ab9af5428f5\": container with ID starting with b84e791012cb4a613860aa9732855ff733f10f51f303f60038cc7ab9af5428f5 not found: ID does not exist" containerID="b84e791012cb4a613860aa9732855ff733f10f51f303f60038cc7ab9af5428f5"
Oct 31 03:18:27.953899 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.953830    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84e791012cb4a613860aa9732855ff733f10f51f303f60038cc7ab9af5428f5"} err="failed to get container status \"b84e791012cb4a613860aa9732855ff733f10f51f303f60038cc7ab9af5428f5\": rpc error: code = NotFound desc = could not find container \"b84e791012cb4a613860aa9732855ff733f10f51f303f60038cc7ab9af5428f5\": container with ID starting with b84e791012cb4a613860aa9732855ff733f10f51f303f60038cc7ab9af5428f5 not found: ID does not exist"
Oct 31 03:18:27.953899 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.953850    4900 scope.go:122] "RemoveContainer" containerID="c4a45d505466fb0959ccfcb115266e429de46a4e657bb848b8229c4566cc654f"
Oct 31 03:18:27.962722 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.962700    4900 scope.go:122] "RemoveContainer" containerID="c4a45d505466fb0959ccfcb115266e429de46a4e657bb848b8229c4566cc654f"
Oct 31 03:18:27.963004 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:27.962987    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a45d505466fb0959ccfcb115266e429de46a4e657bb848b8229c4566cc654f\": container with ID starting with c4a45d505466fb0959ccfcb115266e429de46a4e657bb848b8229c4566cc654f not found: ID does not exist" containerID="c4a45d505466fb0959ccfcb115266e429de46a4e657bb848b8229c4566cc654f"
Oct 31 03:18:27.963104 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:27.963012    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a45d505466fb0959ccfcb115266e429de46a4e657bb848b8229c4566cc654f"} err="failed to get container status \"c4a45d505466fb0959ccfcb115266e429de46a4e657bb848b8229c4566cc654f\": rpc error: code = NotFound desc = could not find container \"c4a45d505466fb0959ccfcb115266e429de46a4e657bb848b8229c4566cc654f\": container with ID starting with c4a45d505466fb0959ccfcb115266e429de46a4e657bb848b8229c4566cc654f not found: ID does not exist"
Oct 31 03:18:28.097548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:28.097458    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"]
Oct 31 03:18:28.143535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:28.143502    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-bac33a89-7d497dd4d7-sqqcg"]
Oct 31 03:18:28.210702 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:28.210673    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"]
Oct 31 03:18:28.212423 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:28.212399    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-bac33a89-c758d7c89-hkhwm"]
Oct 31 03:18:28.992069 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:28.991982    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:29.152451 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.152418    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f60debc-d81d-44ab-807e-c6e3418a6db8" path="/var/lib/kubelet/pods/6f60debc-d81d-44ab-807e-c6e3418a6db8/volumes"
Oct 31 03:18:29.152868 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.152853    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76a019c6-1460-4dc1-b322-1a8e7afa4aed" path="/var/lib/kubelet/pods/76a019c6-1460-4dc1-b322-1a8e7afa4aed/volumes"
Oct 31 03:18:29.460649 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.460573    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"]
Oct 31 03:18:29.461050 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.460777    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l" podUID="f3051a5f-94f1-4a26-a927-d8bd71d1590a" containerName="maas-api" containerID="cri-o://2595af3a0903c6b66ed8925d73cb8b3fab9b8b9ab9e9fe715ff1c5377bf51a1f" gracePeriod=30
Oct 31 03:18:29.634308 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.632246    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:29.717515 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.717453    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:18:29.884283 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.884252    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-maas-api-tls\") pod \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") "
Oct 31 03:18:29.884469 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.884358    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-metrics-certs\") pod \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") "
Oct 31 03:18:29.884469 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.884410    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3051a5f-94f1-4a26-a927-d8bd71d1590a-kube-api-access-p8c75\" (UniqueName: \"kubernetes.io/projected/f3051a5f-94f1-4a26-a927-d8bd71d1590a-kube-api-access-p8c75\") pod \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") "
Oct 31 03:18:29.884469 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.884433    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3051a5f-94f1-4a26-a927-d8bd71d1590a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3051a5f-94f1-4a26-a927-d8bd71d1590a-openshift-service-ca\") pod \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\" (UID: \"f3051a5f-94f1-4a26-a927-d8bd71d1590a\") "
Oct 31 03:18:29.884864 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.884831    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3051a5f-94f1-4a26-a927-d8bd71d1590a-openshift-service-ca" pod "f3051a5f-94f1-4a26-a927-d8bd71d1590a" (UID: "f3051a5f-94f1-4a26-a927-d8bd71d1590a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:29.886410 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.886380    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-maas-api-tls" pod "f3051a5f-94f1-4a26-a927-d8bd71d1590a" (UID: "f3051a5f-94f1-4a26-a927-d8bd71d1590a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:18:29.886410 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.886396    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3051a5f-94f1-4a26-a927-d8bd71d1590a-kube-api-access-p8c75" pod "f3051a5f-94f1-4a26-a927-d8bd71d1590a" (UID: "f3051a5f-94f1-4a26-a927-d8bd71d1590a"). InnerVolumeSpecName "kube-api-access-p8c75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:29.886538 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.886397    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-metrics-certs" pod "f3051a5f-94f1-4a26-a927-d8bd71d1590a" (UID: "f3051a5f-94f1-4a26-a927-d8bd71d1590a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:18:29.958880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.958851    4900 generic.go:356] "Generic (PLEG): container finished" podID="f3051a5f-94f1-4a26-a927-d8bd71d1590a" containerID="2595af3a0903c6b66ed8925d73cb8b3fab9b8b9ab9e9fe715ff1c5377bf51a1f" exitCode=0
Oct 31 03:18:29.959043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.958900    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l" event={"ID":"f3051a5f-94f1-4a26-a927-d8bd71d1590a","Type":"ContainerDied","Data":"2595af3a0903c6b66ed8925d73cb8b3fab9b8b9ab9e9fe715ff1c5377bf51a1f"}
Oct 31 03:18:29.959043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.958904    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"
Oct 31 03:18:29.959043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.958938    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l" event={"ID":"f3051a5f-94f1-4a26-a927-d8bd71d1590a","Type":"ContainerDied","Data":"90781c5968b096fbfc7fe229154956b9f74499cd3dd821f0ab10d361891b4d23"}
Oct 31 03:18:29.959043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.958958    4900 scope.go:122] "RemoveContainer" containerID="2595af3a0903c6b66ed8925d73cb8b3fab9b8b9ab9e9fe715ff1c5377bf51a1f"
Oct 31 03:18:29.967597 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.967561    4900 scope.go:122] "RemoveContainer" containerID="2595af3a0903c6b66ed8925d73cb8b3fab9b8b9ab9e9fe715ff1c5377bf51a1f"
Oct 31 03:18:29.967926 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:29.967904    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2595af3a0903c6b66ed8925d73cb8b3fab9b8b9ab9e9fe715ff1c5377bf51a1f\": container with ID starting with 2595af3a0903c6b66ed8925d73cb8b3fab9b8b9ab9e9fe715ff1c5377bf51a1f not found: ID does not exist" containerID="2595af3a0903c6b66ed8925d73cb8b3fab9b8b9ab9e9fe715ff1c5377bf51a1f"
Oct 31 03:18:29.968008 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.967937    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2595af3a0903c6b66ed8925d73cb8b3fab9b8b9ab9e9fe715ff1c5377bf51a1f"} err="failed to get container status \"2595af3a0903c6b66ed8925d73cb8b3fab9b8b9ab9e9fe715ff1c5377bf51a1f\": rpc error: code = NotFound desc = could not find container \"2595af3a0903c6b66ed8925d73cb8b3fab9b8b9ab9e9fe715ff1c5377bf51a1f\": container with ID starting with 2595af3a0903c6b66ed8925d73cb8b3fab9b8b9ab9e9fe715ff1c5377bf51a1f not found: ID does not exist"
Oct 31 03:18:29.985988 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.985966    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:29.986061 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.985990    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8c75\" (UniqueName: \"kubernetes.io/projected/f3051a5f-94f1-4a26-a927-d8bd71d1590a-kube-api-access-p8c75\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:29.986061 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.986004    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3051a5f-94f1-4a26-a927-d8bd71d1590a-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:29.986061 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:29.986012    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3051a5f-94f1-4a26-a927-d8bd71d1590a-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:30.110878 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:30.110839    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"]
Oct 31 03:18:30.115077 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:30.115052    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4072c5d1-7b99b8f6bf-cb76l"]
Oct 31 03:18:30.734617 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:30.734581    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"]
Oct 31 03:18:30.735084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:30.734769    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67" podUID="73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f" containerName="maas-api" containerID="cri-o://c4a8b295869a8c1fba15549a3068bbeb16bd119cccdae16a0a0563e87c734588" gracePeriod=30
Oct 31 03:18:30.916634 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:30.916577    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:30.967730 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:30.967697    4900 generic.go:356] "Generic (PLEG): container finished" podID="73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f" containerID="c4a8b295869a8c1fba15549a3068bbeb16bd119cccdae16a0a0563e87c734588" exitCode=0
Oct 31 03:18:30.967969 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:30.967778    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67" event={"ID":"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f","Type":"ContainerDied","Data":"c4a8b295869a8c1fba15549a3068bbeb16bd119cccdae16a0a0563e87c734588"}
Oct 31 03:18:30.992228 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:30.992172    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:18:31.096431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.096400    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-openshift-service-ca\") pod \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") "
Oct 31 03:18:31.096622 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.096513    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-metrics-certs\") pod \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") "
Oct 31 03:18:31.096622 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.096538    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-maas-api-tls\") pod \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") "
Oct 31 03:18:31.096622 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.096592    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-kube-api-access-j8856\" (UniqueName: \"kubernetes.io/projected/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-kube-api-access-j8856\") pod \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\" (UID: \"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f\") "
Oct 31 03:18:31.096884 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.096850    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-openshift-service-ca" pod "73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f" (UID: "73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:31.097024 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.096987    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:31.099500 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.099417    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-kube-api-access-j8856" pod "73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f" (UID: "73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f"). InnerVolumeSpecName "kube-api-access-j8856". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:31.099500 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.099459    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-metrics-certs" pod "73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f" (UID: "73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:18:31.099500 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.099458    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-maas-api-tls" pod "73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f" (UID: "73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:18:31.151924 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.151893    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3051a5f-94f1-4a26-a927-d8bd71d1590a" path="/var/lib/kubelet/pods/f3051a5f-94f1-4a26-a927-d8bd71d1590a/volumes"
Oct 31 03:18:31.197713 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.197684    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:31.197713 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.197709    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:31.197713 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.197720    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8856\" (UniqueName: \"kubernetes.io/projected/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f-kube-api-access-j8856\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:31.287104 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.286999    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:31.569294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.569194    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:31.922603 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.922488    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:31.982013 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.981969    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67" event={"ID":"73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f","Type":"ContainerDied","Data":"7954825ee486a105d9917b44983e7f2e71b462aa316b086106fa4992304251e4"}
Oct 31 03:18:31.982013 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.981994    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"
Oct 31 03:18:31.982264 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:31.982055    4900 scope.go:122] "RemoveContainer" containerID="c4a8b295869a8c1fba15549a3068bbeb16bd119cccdae16a0a0563e87c734588"
Oct 31 03:18:32.136519 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:32.136483    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"]
Oct 31 03:18:32.182519 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:32.182443    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-bac33a89-8674897cbd-4ww67"]
Oct 31 03:18:32.477072 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:32.476987    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:32.718530 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:32.718498    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:32.929317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:32.929283    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:33.151462 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:33.151428    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f" path="/var/lib/kubelet/pods/73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f/volumes"
Oct 31 03:18:33.228242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:33.228160    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:33.512881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:33.512850    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:33.750042 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:33.750008    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:33.981212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:33.981131    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:34.669916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.669879    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"]
Oct 31 03:18:34.670302 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.670289    4900 cpu_manager.go:414] "containerMap: removing container" podUID="73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f" containerName="maas-api"
Oct 31 03:18:34.670353 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.670303    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f" containerName="maas-api"
Oct 31 03:18:34.670353 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.670319    4900 cpu_manager.go:414] "containerMap: removing container" podUID="f3051a5f-94f1-4a26-a927-d8bd71d1590a" containerName="maas-api"
Oct 31 03:18:34.670353 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.670324    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3051a5f-94f1-4a26-a927-d8bd71d1590a" containerName="maas-api"
Oct 31 03:18:34.670353 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.670335    4900 cpu_manager.go:414] "containerMap: removing container" podUID="76a019c6-1460-4dc1-b322-1a8e7afa4aed" containerName="payload-pre-processing"
Oct 31 03:18:34.670353 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.670340    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76a019c6-1460-4dc1-b322-1a8e7afa4aed" containerName="payload-pre-processing"
Oct 31 03:18:34.670518 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.670357    4900 cpu_manager.go:414] "containerMap: removing container" podUID="6f60debc-d81d-44ab-807e-c6e3418a6db8" containerName="payload-processing"
Oct 31 03:18:34.670518 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.670362    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f60debc-d81d-44ab-807e-c6e3418a6db8" containerName="payload-processing"
Oct 31 03:18:34.670518 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.670419    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3051a5f-94f1-4a26-a927-d8bd71d1590a" containerName="maas-api"
Oct 31 03:18:34.670518 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.670427    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="76a019c6-1460-4dc1-b322-1a8e7afa4aed" containerName="payload-pre-processing"
Oct 31 03:18:34.670518 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.670437    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f60debc-d81d-44ab-807e-c6e3418a6db8" containerName="payload-processing"
Oct 31 03:18:34.670518 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.670445    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="73fa9c5e-fe56-4f7f-8a57-e7a5dadda41f" containerName="maas-api"
Oct 31 03:18:34.674757 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.674737    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.748970 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.748940    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"]
Oct 31 03:18:34.760200 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.760174    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:34.772412 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.772386    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-dd5b85-openshift-default-dockercfg-b24df\""
Oct 31 03:18:34.845507 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.845473    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.845671 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.845532    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.845671 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.845620    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x846w\" (UniqueName: \"kubernetes.io/projected/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-kube-api-access-x846w\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.845741 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.845667    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.845741 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.845710    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.845833 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.845783    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.845878 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.845862    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.845916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.845894    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.845949 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.845916    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.947156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.947074    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.947156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.947132    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.947156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.947154    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.947425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.947176    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.947425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.947198    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.947425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.947232    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.947425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.947308    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.947425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.947337    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x846w\" (UniqueName: \"kubernetes.io/projected/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-kube-api-access-x846w\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.947425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.947383    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.947803 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.947580    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.947803 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.947641    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.947803 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.947669    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.947803 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.947726    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.948103 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.948085    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.949535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.949516    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.949773 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.949752    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:34.986651 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:34.986617    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:35.001416 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:35.001382    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:35.001563 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:35.001396    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x846w\" (UniqueName: \"kubernetes.io/projected/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-kube-api-access-x846w\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:35.206595 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:35.206512    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:35.286961 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:35.286928    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:35.750641 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:35.750609    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"]
Oct 31 03:18:35.752111 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:18:35.752079    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab28c8e8_eb9f_41f2_b7f7_59c6c6a2d1a8.slice/crio-c142e37ac4f9bae48968bf0678de7ab4b6aa77f3c82ce4f9aa474387cbd2bce8 WatchSource:0}: Error finding container c142e37ac4f9bae48968bf0678de7ab4b6aa77f3c82ce4f9aa474387cbd2bce8: Status 404 returned error can't find the container with id c142e37ac4f9bae48968bf0678de7ab4b6aa77f3c82ce4f9aa474387cbd2bce8
Oct 31 03:18:35.754421 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:35.754391    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:18:35.754504 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:35.754457    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:18:35.754504 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:35.754487    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:18:36.007241 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:36.007156    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs" event={"ID":"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8","Type":"ContainerStarted","Data":"98077e4376b1c2336b924aff4b23068ecad4a75e30a153ae0b0650b0453defa5"}
Oct 31 03:18:36.007241 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:36.007219    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs" event={"ID":"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8","Type":"ContainerStarted","Data":"c142e37ac4f9bae48968bf0678de7ab4b6aa77f3c82ce4f9aa474387cbd2bce8"}
Oct 31 03:18:36.120073 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:36.119575    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs" podStartSLOduration=2.119557693 podStartE2EDuration="2.119557693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:36.118720658 +0000 UTC m=+669.085054036" watchObservedRunningTime="2026-10-31 03:18:36.119557693 +0000 UTC m=+669.085891039"
Oct 31 03:18:36.287605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:36.287518    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:36.292615 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:36.292589    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:36.983588 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:36.983550    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:18:37.011654 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.011622    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:37.012694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.012677    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:18:37.433204 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.433124    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"]
Oct 31 03:18:37.433347 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.433326    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r" podUID="7d4a653b-999b-4878-aaae-c29887c4bcab" containerName="maas-api" containerID="cri-o://b25f5fddd45496ea102569d347c8e27270dc7a823ef5e3b084780d7e6751abbe" gracePeriod=30
Oct 31 03:18:37.613645 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.613615    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:37.691234 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.691174    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:18:37.772880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.772837    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d4a653b-999b-4878-aaae-c29887c4bcab-kube-api-access-rpb72\" (UniqueName: \"kubernetes.io/projected/7d4a653b-999b-4878-aaae-c29887c4bcab-kube-api-access-rpb72\") pod \"7d4a653b-999b-4878-aaae-c29887c4bcab\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") "
Oct 31 03:18:37.773033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.772932    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-metrics-certs\") pod \"7d4a653b-999b-4878-aaae-c29887c4bcab\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") "
Oct 31 03:18:37.773033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.773020    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-maas-api-tls\") pod \"7d4a653b-999b-4878-aaae-c29887c4bcab\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") "
Oct 31 03:18:37.773131 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.773050    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d4a653b-999b-4878-aaae-c29887c4bcab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4a653b-999b-4878-aaae-c29887c4bcab-openshift-service-ca\") pod \"7d4a653b-999b-4878-aaae-c29887c4bcab\" (UID: \"7d4a653b-999b-4878-aaae-c29887c4bcab\") "
Oct 31 03:18:37.773491 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.773458    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4a653b-999b-4878-aaae-c29887c4bcab-openshift-service-ca" pod "7d4a653b-999b-4878-aaae-c29887c4bcab" (UID: "7d4a653b-999b-4878-aaae-c29887c4bcab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:37.774975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.774953    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-metrics-certs" pod "7d4a653b-999b-4878-aaae-c29887c4bcab" (UID: "7d4a653b-999b-4878-aaae-c29887c4bcab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:18:37.775067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.775029    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4a653b-999b-4878-aaae-c29887c4bcab-kube-api-access-rpb72" pod "7d4a653b-999b-4878-aaae-c29887c4bcab" (UID: "7d4a653b-999b-4878-aaae-c29887c4bcab"). InnerVolumeSpecName "kube-api-access-rpb72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:37.775067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.775055    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-maas-api-tls" pod "7d4a653b-999b-4878-aaae-c29887c4bcab" (UID: "7d4a653b-999b-4878-aaae-c29887c4bcab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:18:37.853072 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.853035    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:37.874424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.874395    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:37.874424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.874423    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d4a653b-999b-4878-aaae-c29887c4bcab-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:37.874619 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.874435    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d4a653b-999b-4878-aaae-c29887c4bcab-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:37.874619 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:37.874445    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpb72\" (UniqueName: \"kubernetes.io/projected/7d4a653b-999b-4878-aaae-c29887c4bcab-kube-api-access-rpb72\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:38.017872 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.017838    4900 generic.go:356] "Generic (PLEG): container finished" podID="7d4a653b-999b-4878-aaae-c29887c4bcab" containerID="b25f5fddd45496ea102569d347c8e27270dc7a823ef5e3b084780d7e6751abbe" exitCode=0
Oct 31 03:18:38.018325 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.017887    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"
Oct 31 03:18:38.018325 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.017924    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r" event={"ID":"7d4a653b-999b-4878-aaae-c29887c4bcab","Type":"ContainerDied","Data":"b25f5fddd45496ea102569d347c8e27270dc7a823ef5e3b084780d7e6751abbe"}
Oct 31 03:18:38.018325 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.017973    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r" event={"ID":"7d4a653b-999b-4878-aaae-c29887c4bcab","Type":"ContainerDied","Data":"76e00853bff8c68dae5870f362dfc89b42ffa84afdb6ffa90fd15e5d83871134"}
Oct 31 03:18:38.018325 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.017999    4900 scope.go:122] "RemoveContainer" containerID="b25f5fddd45496ea102569d347c8e27270dc7a823ef5e3b084780d7e6751abbe"
Oct 31 03:18:38.027124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.027109    4900 scope.go:122] "RemoveContainer" containerID="b25f5fddd45496ea102569d347c8e27270dc7a823ef5e3b084780d7e6751abbe"
Oct 31 03:18:38.027372 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:38.027353    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25f5fddd45496ea102569d347c8e27270dc7a823ef5e3b084780d7e6751abbe\": container with ID starting with b25f5fddd45496ea102569d347c8e27270dc7a823ef5e3b084780d7e6751abbe not found: ID does not exist" containerID="b25f5fddd45496ea102569d347c8e27270dc7a823ef5e3b084780d7e6751abbe"
Oct 31 03:18:38.027438 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.027381    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25f5fddd45496ea102569d347c8e27270dc7a823ef5e3b084780d7e6751abbe"} err="failed to get container status \"b25f5fddd45496ea102569d347c8e27270dc7a823ef5e3b084780d7e6751abbe\": rpc error: code = NotFound desc = could not find container \"b25f5fddd45496ea102569d347c8e27270dc7a823ef5e3b084780d7e6751abbe\": container with ID starting with b25f5fddd45496ea102569d347c8e27270dc7a823ef5e3b084780d7e6751abbe not found: ID does not exist"
Oct 31 03:18:38.175006 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.174967    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"]
Oct 31 03:18:38.220462 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.220427    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2f427162-645fbcdbcc-kwh5r"]
Oct 31 03:18:38.678678 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.678638    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"]
Oct 31 03:18:38.678890 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.678843    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj" podUID="b9a24e69-35fb-45c5-a85e-14d410aa89dc" containerName="maas-api" containerID="cri-o://d27d26037dc5ba5dcea843031233446e398cf0d205059571de7ab818090924f3" gracePeriod=30
Oct 31 03:18:38.862983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.862949    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:38.922475 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.922452    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:18:38.985706 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.985667    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9a24e69-35fb-45c5-a85e-14d410aa89dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a24e69-35fb-45c5-a85e-14d410aa89dc-openshift-service-ca\") pod \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") "
Oct 31 03:18:38.985894 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.985769    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-maas-api-tls\") pod \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") "
Oct 31 03:18:38.985894 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.985816    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a24e69-35fb-45c5-a85e-14d410aa89dc-kube-api-access-kjln2\" (UniqueName: \"kubernetes.io/projected/b9a24e69-35fb-45c5-a85e-14d410aa89dc-kube-api-access-kjln2\") pod \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") "
Oct 31 03:18:38.985894 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.985877    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-metrics-certs\") pod \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\" (UID: \"b9a24e69-35fb-45c5-a85e-14d410aa89dc\") "
Oct 31 03:18:38.986136 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.986105    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a24e69-35fb-45c5-a85e-14d410aa89dc-openshift-service-ca" pod "b9a24e69-35fb-45c5-a85e-14d410aa89dc" (UID: "b9a24e69-35fb-45c5-a85e-14d410aa89dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:38.986509 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.986476    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a24e69-35fb-45c5-a85e-14d410aa89dc-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:38.988155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.988125    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-metrics-certs" pod "b9a24e69-35fb-45c5-a85e-14d410aa89dc" (UID: "b9a24e69-35fb-45c5-a85e-14d410aa89dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:18:38.988449 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.988423    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-maas-api-tls" pod "b9a24e69-35fb-45c5-a85e-14d410aa89dc" (UID: "b9a24e69-35fb-45c5-a85e-14d410aa89dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:18:38.988449 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:38.988429    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a24e69-35fb-45c5-a85e-14d410aa89dc-kube-api-access-kjln2" pod "b9a24e69-35fb-45c5-a85e-14d410aa89dc" (UID: "b9a24e69-35fb-45c5-a85e-14d410aa89dc"). InnerVolumeSpecName "kube-api-access-kjln2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:39.024849 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.024814    4900 generic.go:356] "Generic (PLEG): container finished" podID="b9a24e69-35fb-45c5-a85e-14d410aa89dc" containerID="d27d26037dc5ba5dcea843031233446e398cf0d205059571de7ab818090924f3" exitCode=0
Oct 31 03:18:39.025368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.024870    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"
Oct 31 03:18:39.025368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.024890    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj" event={"ID":"b9a24e69-35fb-45c5-a85e-14d410aa89dc","Type":"ContainerDied","Data":"d27d26037dc5ba5dcea843031233446e398cf0d205059571de7ab818090924f3"}
Oct 31 03:18:39.025368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.024978    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj" event={"ID":"b9a24e69-35fb-45c5-a85e-14d410aa89dc","Type":"ContainerDied","Data":"6f0e1f823618c6407eb6abce633cc17eb5a1fbc633c0addc931ac3a5e9fd9c1c"}
Oct 31 03:18:39.025368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.025002    4900 scope.go:122] "RemoveContainer" containerID="d27d26037dc5ba5dcea843031233446e398cf0d205059571de7ab818090924f3"
Oct 31 03:18:39.034917 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.034903    4900 scope.go:122] "RemoveContainer" containerID="d27d26037dc5ba5dcea843031233446e398cf0d205059571de7ab818090924f3"
Oct 31 03:18:39.035205 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:39.035186    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27d26037dc5ba5dcea843031233446e398cf0d205059571de7ab818090924f3\": container with ID starting with d27d26037dc5ba5dcea843031233446e398cf0d205059571de7ab818090924f3 not found: ID does not exist" containerID="d27d26037dc5ba5dcea843031233446e398cf0d205059571de7ab818090924f3"
Oct 31 03:18:39.035252 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.035213    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27d26037dc5ba5dcea843031233446e398cf0d205059571de7ab818090924f3"} err="failed to get container status \"d27d26037dc5ba5dcea843031233446e398cf0d205059571de7ab818090924f3\": rpc error: code = NotFound desc = could not find container \"d27d26037dc5ba5dcea843031233446e398cf0d205059571de7ab818090924f3\": container with ID starting with d27d26037dc5ba5dcea843031233446e398cf0d205059571de7ab818090924f3 not found: ID does not exist"
Oct 31 03:18:39.087806 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.087756    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:39.087806 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.087804    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a24e69-35fb-45c5-a85e-14d410aa89dc-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:39.088011 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.087825    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjln2\" (UniqueName: \"kubernetes.io/projected/b9a24e69-35fb-45c5-a85e-14d410aa89dc-kube-api-access-kjln2\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:39.152509 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.152469    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d4a653b-999b-4878-aaae-c29887c4bcab" path="/var/lib/kubelet/pods/7d4a653b-999b-4878-aaae-c29887c4bcab/volumes"
Oct 31 03:18:39.152867 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.152850    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"]
Oct 31 03:18:39.156043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.156017    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15000fe0-6847dcbf5-ctssj"]
Oct 31 03:18:39.265467 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.265380    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:39.493488 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.493458    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:39.704627 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:39.704546    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:40.973723 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:40.973684    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:41.151480 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:41.151447    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a24e69-35fb-45c5-a85e-14d410aa89dc" path="/var/lib/kubelet/pods/b9a24e69-35fb-45c5-a85e-14d410aa89dc/volumes"
Oct 31 03:18:41.239458 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:41.239432    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"]
Oct 31 03:18:41.239656 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:41.239625    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j" podUID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" containerName="istio-proxy" containerID="cri-o://bd98d8a280b09865c0962225f7009c47bef57b5ed987e4819bbd290606d7f65f" gracePeriod=30
Oct 31 03:18:41.326095 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:41.326061    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:41.546504 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:41.546432    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:41.771411 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:41.771377    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:42.094075 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:42.094038    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:42.693502 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:42.693473    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:42.904846 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:42.904819    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:44.356402 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:44.356351    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:44.654202 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:44.654107    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:44.946426 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:44.946329    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:45.003168 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:45.003135    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"]
Oct 31 03:18:45.003366 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:45.003346    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb" podUID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerName="istio-proxy" containerID="cri-o://1dd83886aa3dee176f719060f34d8140a6d2aad40f223c770e127bceda7cfe17" gracePeriod=30
Oct 31 03:18:45.189699 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:45.189666    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:45.400833 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:45.400771    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:45.404577 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:45.404550    4900 patch_prober.go:29] interesting pod/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:18:45.404681 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:45.404605    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb" podUID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:18:45.801279 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:45.801246    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:46.389598 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.389561    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:46.493423 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.493399    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:18:46.565307 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.564754    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istiod-ca-cert\") pod \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") "
Oct 31 03:18:46.565307 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.565212    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istiod-ca-cert" pod "0cb621eb-a9f8-4a9e-8881-82d61fe283f6" (UID: "0cb621eb-a9f8-4a9e-8881-82d61fe283f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:46.565527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.565319    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-podinfo\") pod \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") "
Oct 31 03:18:46.565527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.565367    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-data\") pod \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") "
Oct 31 03:18:46.565527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.565500    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-kube-api-access-6dsst\" (UniqueName: \"kubernetes.io/projected/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-kube-api-access-6dsst\") pod \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") "
Oct 31 03:18:46.565638 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.565539    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-credential-socket\") pod \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") "
Oct 31 03:18:46.565638 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.565576    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-token\" (UniqueName: \"kubernetes.io/projected/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-token\") pod \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") "
Oct 31 03:18:46.565638 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.565623    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-workload-certs\") pod \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") "
Oct 31 03:18:46.565836 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.565678    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-workload-socket\") pod \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") "
Oct 31 03:18:46.565836 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.565712    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-envoy\") pod \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\" (UID: \"0cb621eb-a9f8-4a9e-8881-82d61fe283f6\") "
Oct 31 03:18:46.566176 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.566148    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:46.567438 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.566724    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-workload-certs" pod "0cb621eb-a9f8-4a9e-8881-82d61fe283f6" (UID: "0cb621eb-a9f8-4a9e-8881-82d61fe283f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:46.567438 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.567212    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-data" pod "0cb621eb-a9f8-4a9e-8881-82d61fe283f6" (UID: "0cb621eb-a9f8-4a9e-8881-82d61fe283f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:46.567438 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.567312    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-credential-socket" pod "0cb621eb-a9f8-4a9e-8881-82d61fe283f6" (UID: "0cb621eb-a9f8-4a9e-8881-82d61fe283f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:46.567438 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.567367    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-workload-socket" pod "0cb621eb-a9f8-4a9e-8881-82d61fe283f6" (UID: "0cb621eb-a9f8-4a9e-8881-82d61fe283f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:46.568709 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.568670    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-podinfo" pod "0cb621eb-a9f8-4a9e-8881-82d61fe283f6" (UID: "0cb621eb-a9f8-4a9e-8881-82d61fe283f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:18:46.569061 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.569042    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-envoy" pod "0cb621eb-a9f8-4a9e-8881-82d61fe283f6" (UID: "0cb621eb-a9f8-4a9e-8881-82d61fe283f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:46.569122 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.569079    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-token" pod "0cb621eb-a9f8-4a9e-8881-82d61fe283f6" (UID: "0cb621eb-a9f8-4a9e-8881-82d61fe283f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:46.569505 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.569489    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-kube-api-access-6dsst" pod "0cb621eb-a9f8-4a9e-8881-82d61fe283f6" (UID: "0cb621eb-a9f8-4a9e-8881-82d61fe283f6"). InnerVolumeSpecName "kube-api-access-6dsst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:46.594779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.594747    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"]
Oct 31 03:18:46.595276 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.595255    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" containerName="istio-proxy"
Oct 31 03:18:46.595276 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.595275    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" containerName="istio-proxy"
Oct 31 03:18:46.595429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.595289    4900 cpu_manager.go:414] "containerMap: removing container" podUID="b9a24e69-35fb-45c5-a85e-14d410aa89dc" containerName="maas-api"
Oct 31 03:18:46.595429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.595297    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a24e69-35fb-45c5-a85e-14d410aa89dc" containerName="maas-api"
Oct 31 03:18:46.595429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.595316    4900 cpu_manager.go:414] "containerMap: removing container" podUID="7d4a653b-999b-4878-aaae-c29887c4bcab" containerName="maas-api"
Oct 31 03:18:46.595429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.595322    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4a653b-999b-4878-aaae-c29887c4bcab" containerName="maas-api"
Oct 31 03:18:46.595429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.595332    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" containerName="istio-proxy"
Oct 31 03:18:46.595429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.595337    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" containerName="istio-proxy"
Oct 31 03:18:46.595429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.595415    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" containerName="istio-proxy"
Oct 31 03:18:46.595429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.595424    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4a653b-999b-4878-aaae-c29887c4bcab" containerName="maas-api"
Oct 31 03:18:46.595429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.595429    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" containerName="istio-proxy"
Oct 31 03:18:46.595693 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.595441    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a24e69-35fb-45c5-a85e-14d410aa89dc" containerName="maas-api"
Oct 31 03:18:46.599018 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.598986    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.637437 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.637405    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"]
Oct 31 03:18:46.653540 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.653515    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d39257-openshift-default-dockercfg-bm5z5\""
Oct 31 03:18:46.666719 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.666690    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwl9r\" (UniqueName: \"kubernetes.io/projected/9b030d08-9d78-4c66-8729-243e64177d52-kube-api-access-hwl9r\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.666860 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.666741    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b030d08-9d78-4c66-8729-243e64177d52-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.666936 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.666865    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b030d08-9d78-4c66-8729-243e64177d52-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.666983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.666933    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b030d08-9d78-4c66-8729-243e64177d52-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.667030 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.667010    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.667064 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.667044    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.667146 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.667131    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.667205 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.667190    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.667252 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.667237    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.667324 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.667311    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:46.667364 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.667330    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:46.667364 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.667346    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dsst\" (UniqueName: \"kubernetes.io/projected/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-kube-api-access-6dsst\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:46.667419 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.667360    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:46.667419 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.667374    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:46.667419 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.667391    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:46.667419 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.667406    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:46.667538 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.667419    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cb621eb-a9f8-4a9e-8881-82d61fe283f6-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:46.734659 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.734615    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:46.768879 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.768845    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.769045 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.768900    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.769045 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.768939    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.769045 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.768975    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwl9r\" (UniqueName: \"kubernetes.io/projected/9b030d08-9d78-4c66-8729-243e64177d52-kube-api-access-hwl9r\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.769045 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.769014    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b030d08-9d78-4c66-8729-243e64177d52-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.769628 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.769141    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b030d08-9d78-4c66-8729-243e64177d52-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.769628 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.769199    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b030d08-9d78-4c66-8729-243e64177d52-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.769628 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.769274    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.769628 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.769317    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.769628 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.769364    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.769628 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.769479    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.769628 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.769519    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.769981 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.769757    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.769981 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.769838    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b030d08-9d78-4c66-8729-243e64177d52-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.771726 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.771707    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.771957 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.771938    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b030d08-9d78-4c66-8729-243e64177d52-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.822110 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.822044    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b030d08-9d78-4c66-8729-243e64177d52-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.822227 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.822147    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwl9r\" (UniqueName: \"kubernetes.io/projected/9b030d08-9d78-4c66-8729-243e64177d52-kube-api-access-hwl9r\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:46.912996 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:46.912955    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:47.062732 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.062698    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:47.065294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.065265    4900 generic.go:356] "Generic (PLEG): container finished" podID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" containerID="bd98d8a280b09865c0962225f7009c47bef57b5ed987e4819bbd290606d7f65f" exitCode=0
Oct 31 03:18:47.065425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.065318    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"
Oct 31 03:18:47.065425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.065347    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j" event={"ID":"0cb621eb-a9f8-4a9e-8881-82d61fe283f6","Type":"ContainerDied","Data":"bd98d8a280b09865c0962225f7009c47bef57b5ed987e4819bbd290606d7f65f"}
Oct 31 03:18:47.065425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.065409    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j" event={"ID":"0cb621eb-a9f8-4a9e-8881-82d61fe283f6","Type":"ContainerDied","Data":"9e3f34de35d93fb7ce94a2531131fdcad06989a3c62d57eb841a726ca83b2af2"}
Oct 31 03:18:47.065621 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.065439    4900 scope.go:122] "RemoveContainer" containerID="bd98d8a280b09865c0962225f7009c47bef57b5ed987e4819bbd290606d7f65f"
Oct 31 03:18:47.077187 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.077168    4900 scope.go:122] "RemoveContainer" containerID="f38b2c230fcde32d5cbd9c8c8fbe6066d71ddb3049eb0e1da287f41bd8bfd6ca"
Oct 31 03:18:47.084888 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.084870    4900 scope.go:122] "RemoveContainer" containerID="bd98d8a280b09865c0962225f7009c47bef57b5ed987e4819bbd290606d7f65f"
Oct 31 03:18:47.085140 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:47.085121    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd98d8a280b09865c0962225f7009c47bef57b5ed987e4819bbd290606d7f65f\": container with ID starting with bd98d8a280b09865c0962225f7009c47bef57b5ed987e4819bbd290606d7f65f not found: ID does not exist" containerID="bd98d8a280b09865c0962225f7009c47bef57b5ed987e4819bbd290606d7f65f"
Oct 31 03:18:47.085184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.085148    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd98d8a280b09865c0962225f7009c47bef57b5ed987e4819bbd290606d7f65f"} err="failed to get container status \"bd98d8a280b09865c0962225f7009c47bef57b5ed987e4819bbd290606d7f65f\": rpc error: code = NotFound desc = could not find container \"bd98d8a280b09865c0962225f7009c47bef57b5ed987e4819bbd290606d7f65f\": container with ID starting with bd98d8a280b09865c0962225f7009c47bef57b5ed987e4819bbd290606d7f65f not found: ID does not exist"
Oct 31 03:18:47.085184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.085165    4900 scope.go:122] "RemoveContainer" containerID="f38b2c230fcde32d5cbd9c8c8fbe6066d71ddb3049eb0e1da287f41bd8bfd6ca"
Oct 31 03:18:47.085363 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:47.085349    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f38b2c230fcde32d5cbd9c8c8fbe6066d71ddb3049eb0e1da287f41bd8bfd6ca\": container with ID starting with f38b2c230fcde32d5cbd9c8c8fbe6066d71ddb3049eb0e1da287f41bd8bfd6ca not found: ID does not exist" containerID="f38b2c230fcde32d5cbd9c8c8fbe6066d71ddb3049eb0e1da287f41bd8bfd6ca"
Oct 31 03:18:47.085406 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.085367    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38b2c230fcde32d5cbd9c8c8fbe6066d71ddb3049eb0e1da287f41bd8bfd6ca"} err="failed to get container status \"f38b2c230fcde32d5cbd9c8c8fbe6066d71ddb3049eb0e1da287f41bd8bfd6ca\": rpc error: code = NotFound desc = could not find container \"f38b2c230fcde32d5cbd9c8c8fbe6066d71ddb3049eb0e1da287f41bd8bfd6ca\": container with ID starting with f38b2c230fcde32d5cbd9c8c8fbe6066d71ddb3049eb0e1da287f41bd8bfd6ca not found: ID does not exist"
Oct 31 03:18:47.224544 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.224512    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"]
Oct 31 03:18:47.234644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.234615    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2f427162-openshift-default-8b8547849-9wg6j"]
Oct 31 03:18:47.378521 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.378497    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"]
Oct 31 03:18:47.380455 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:18:47.380418    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b030d08_9d78_4c66_8729_243e64177d52.slice/crio-5b5774dc109adfd9de9482e685ba730f0bb5d182abd939aac4ea5665143e14d3 WatchSource:0}: Error finding container 5b5774dc109adfd9de9482e685ba730f0bb5d182abd939aac4ea5665143e14d3: Status 404 returned error can't find the container with id 5b5774dc109adfd9de9482e685ba730f0bb5d182abd939aac4ea5665143e14d3
Oct 31 03:18:47.382822 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.382781    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:18:47.382942 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.382855    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:18:47.382942 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.382888    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:18:47.403286 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.403262    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:47.619956 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.619921    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:47.717462 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.717380    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"]
Oct 31 03:18:47.721745 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.721724    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"
Oct 31 03:18:47.771151 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.771122    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-dd5b85\""
Oct 31 03:18:47.777092 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.777062    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"]
Oct 31 03:18:47.796248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.796223    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfv9l\" (UniqueName: \"kubernetes.io/projected/25920611-22a3-431c-b9cd-140eddaee0d0-kube-api-access-jfv9l\") pod \"payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9\" (UID: \"25920611-22a3-431c-b9cd-140eddaee0d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"
Oct 31 03:18:47.796397 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.796273    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25920611-22a3-431c-b9cd-140eddaee0d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9\" (UID: \"25920611-22a3-431c-b9cd-140eddaee0d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"
Oct 31 03:18:47.835911 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.835873    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-dd5b85-dockercfg-gnwln\""
Oct 31 03:18:47.897510 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.897472    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfv9l\" (UniqueName: \"kubernetes.io/projected/25920611-22a3-431c-b9cd-140eddaee0d0-kube-api-access-jfv9l\") pod \"payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9\" (UID: \"25920611-22a3-431c-b9cd-140eddaee0d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"
Oct 31 03:18:47.897670 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.897539    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25920611-22a3-431c-b9cd-140eddaee0d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9\" (UID: \"25920611-22a3-431c-b9cd-140eddaee0d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"
Oct 31 03:18:47.898270 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.898248    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25920611-22a3-431c-b9cd-140eddaee0d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9\" (UID: \"25920611-22a3-431c-b9cd-140eddaee0d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"
Oct 31 03:18:47.957481 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:47.957430    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfv9l\" (UniqueName: \"kubernetes.io/projected/25920611-22a3-431c-b9cd-140eddaee0d0-kube-api-access-jfv9l\") pod \"payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9\" (UID: \"25920611-22a3-431c-b9cd-140eddaee0d0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"
Oct 31 03:18:48.034637 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:48.034599    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"
Oct 31 03:18:48.073429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:48.073389    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" event={"ID":"9b030d08-9d78-4c66-8729-243e64177d52","Type":"ContainerStarted","Data":"9b0a944a28df52f75b3d7b0d87d896d03519463561f8948d6f52023db2e66149"}
Oct 31 03:18:48.073615 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:48.073457    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" event={"ID":"9b030d08-9d78-4c66-8729-243e64177d52","Type":"ContainerStarted","Data":"5b5774dc109adfd9de9482e685ba730f0bb5d182abd939aac4ea5665143e14d3"}
Oct 31 03:18:48.191314 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:48.191263    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" podStartSLOduration=2.191250672 podStartE2EDuration="2.191250672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:48.189328568 +0000 UTC m=+681.155661912" watchObservedRunningTime="2026-10-31 03:18:48.191250672 +0000 UTC m=+681.157584069"
Oct 31 03:18:48.334374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:48.334242    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:48.400077 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:48.400051    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"]
Oct 31 03:18:48.402448 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:18:48.402420    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25920611_22a3_431c_b9cd_140eddaee0d0.slice/crio-d5d8919cee580c756bd48ff9ded609e8e7436c6e1201ed8885f691823cbd0997 WatchSource:0}: Error finding container d5d8919cee580c756bd48ff9ded609e8e7436c6e1201ed8885f691823cbd0997: Status 404 returned error can't find the container with id d5d8919cee580c756bd48ff9ded609e8e7436c6e1201ed8885f691823cbd0997
Oct 31 03:18:48.569013 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:48.568983    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:48.913819 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:48.913764    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:48.920204 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:48.920179    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:48.936521 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:48.936489    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:49.080193 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:49.080157    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9" event={"ID":"25920611-22a3-431c-b9cd-140eddaee0d0","Type":"ContainerStarted","Data":"8f5d6dd003f40db5fe7b43bb53682ed56b125e6820269f7ca824557cc56f73f9"}
Oct 31 03:18:49.080380 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:49.080321    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"
Oct 31 03:18:49.080380 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:49.080361    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:49.080467 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:49.080383    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9" event={"ID":"25920611-22a3-431c-b9cd-140eddaee0d0","Type":"ContainerStarted","Data":"d5d8919cee580c756bd48ff9ded609e8e7436c6e1201ed8885f691823cbd0997"}
Oct 31 03:18:49.081198 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:49.081181    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:18:49.150508 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:49.150473    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:49.151320 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:49.151300    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb621eb-a9f8-4a9e-8881-82d61fe283f6" path="/var/lib/kubelet/pods/0cb621eb-a9f8-4a9e-8881-82d61fe283f6/volumes"
Oct 31 03:18:49.197609 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:49.197512    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9" podStartSLOduration=2.197498698 podStartE2EDuration="2.197498698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:49.194350936 +0000 UTC m=+682.160684279" watchObservedRunningTime="2026-10-31 03:18:49.197498698 +0000 UTC m=+682.163832042"
Oct 31 03:18:50.262135 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.262112    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:18:50.325880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.325770    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-envoy\") pod \"195f8e1b-34bf-41ff-a708-17364e8a42fd\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") "
Oct 31 03:18:50.326027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.325891    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-podinfo\") pod \"195f8e1b-34bf-41ff-a708-17364e8a42fd\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") "
Oct 31 03:18:50.326027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.325928    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-workload-certs\") pod \"195f8e1b-34bf-41ff-a708-17364e8a42fd\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") "
Oct 31 03:18:50.326027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.325971    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-data\") pod \"195f8e1b-34bf-41ff-a708-17364e8a42fd\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") "
Oct 31 03:18:50.326027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.326018    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195f8e1b-34bf-41ff-a708-17364e8a42fd-kube-api-access-tcw6j\" (UniqueName: \"kubernetes.io/projected/195f8e1b-34bf-41ff-a708-17364e8a42fd-kube-api-access-tcw6j\") pod \"195f8e1b-34bf-41ff-a708-17364e8a42fd\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") "
Oct 31 03:18:50.326252 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.326080    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-credential-socket\") pod \"195f8e1b-34bf-41ff-a708-17364e8a42fd\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") "
Oct 31 03:18:50.326252 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.326142    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-workload-socket\") pod \"195f8e1b-34bf-41ff-a708-17364e8a42fd\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") "
Oct 31 03:18:50.326252 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.326210    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/195f8e1b-34bf-41ff-a708-17364e8a42fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195f8e1b-34bf-41ff-a708-17364e8a42fd-istiod-ca-cert\") pod \"195f8e1b-34bf-41ff-a708-17364e8a42fd\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") "
Oct 31 03:18:50.326401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.326258    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-token\" (UniqueName: \"kubernetes.io/projected/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-token\") pod \"195f8e1b-34bf-41ff-a708-17364e8a42fd\" (UID: \"195f8e1b-34bf-41ff-a708-17364e8a42fd\") "
Oct 31 03:18:50.326401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.326282    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-workload-certs" pod "195f8e1b-34bf-41ff-a708-17364e8a42fd" (UID: "195f8e1b-34bf-41ff-a708-17364e8a42fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:50.326401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.326327    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-data" pod "195f8e1b-34bf-41ff-a708-17364e8a42fd" (UID: "195f8e1b-34bf-41ff-a708-17364e8a42fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:50.326556 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.326517    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-workload-socket" pod "195f8e1b-34bf-41ff-a708-17364e8a42fd" (UID: "195f8e1b-34bf-41ff-a708-17364e8a42fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:50.326717 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.326620    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195f8e1b-34bf-41ff-a708-17364e8a42fd-istiod-ca-cert" pod "195f8e1b-34bf-41ff-a708-17364e8a42fd" (UID: "195f8e1b-34bf-41ff-a708-17364e8a42fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:50.326717 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.326649    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:50.326717 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.326666    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:50.326717 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.326675    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:50.326717 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.326703    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-credential-socket" pod "195f8e1b-34bf-41ff-a708-17364e8a42fd" (UID: "195f8e1b-34bf-41ff-a708-17364e8a42fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:50.328076 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.328049    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-envoy" pod "195f8e1b-34bf-41ff-a708-17364e8a42fd" (UID: "195f8e1b-34bf-41ff-a708-17364e8a42fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:50.328187 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.328093    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-podinfo" pod "195f8e1b-34bf-41ff-a708-17364e8a42fd" (UID: "195f8e1b-34bf-41ff-a708-17364e8a42fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:18:50.328225 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.328185    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195f8e1b-34bf-41ff-a708-17364e8a42fd-kube-api-access-tcw6j" pod "195f8e1b-34bf-41ff-a708-17364e8a42fd" (UID: "195f8e1b-34bf-41ff-a708-17364e8a42fd"). InnerVolumeSpecName "kube-api-access-tcw6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:50.328468 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.328446    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-token" pod "195f8e1b-34bf-41ff-a708-17364e8a42fd" (UID: "195f8e1b-34bf-41ff-a708-17364e8a42fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:50.428328 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.428293    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195f8e1b-34bf-41ff-a708-17364e8a42fd-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:50.428328 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.428331    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:50.428501 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.428346    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:50.428501 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.428359    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195f8e1b-34bf-41ff-a708-17364e8a42fd-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:50.428501 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.428374    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcw6j\" (UniqueName: \"kubernetes.io/projected/195f8e1b-34bf-41ff-a708-17364e8a42fd-kube-api-access-tcw6j\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:50.428501 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:50.428389    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195f8e1b-34bf-41ff-a708-17364e8a42fd-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:51.098621 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.098579    4900 generic.go:356] "Generic (PLEG): container finished" podID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerID="1dd83886aa3dee176f719060f34d8140a6d2aad40f223c770e127bceda7cfe17" exitCode=0
Oct 31 03:18:51.098810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.098648    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"
Oct 31 03:18:51.098810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.098661    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb" event={"ID":"195f8e1b-34bf-41ff-a708-17364e8a42fd","Type":"ContainerDied","Data":"1dd83886aa3dee176f719060f34d8140a6d2aad40f223c770e127bceda7cfe17"}
Oct 31 03:18:51.098810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.098752    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb" event={"ID":"195f8e1b-34bf-41ff-a708-17364e8a42fd","Type":"ContainerDied","Data":"38b5602eeff28a6418ad2538d2222dedde04c549b1ac25af1ce64951b28915b5"}
Oct 31 03:18:51.098810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.098782    4900 scope.go:122] "RemoveContainer" containerID="1dd83886aa3dee176f719060f34d8140a6d2aad40f223c770e127bceda7cfe17"
Oct 31 03:18:51.109674 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.109655    4900 scope.go:122] "RemoveContainer" containerID="05ad7cde6fed2c5b32485c1f6f5255cd2173a788bbae64d208a4c279a381d00d"
Oct 31 03:18:51.118008 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.117858    4900 scope.go:122] "RemoveContainer" containerID="1dd83886aa3dee176f719060f34d8140a6d2aad40f223c770e127bceda7cfe17"
Oct 31 03:18:51.118113 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:51.118091    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd83886aa3dee176f719060f34d8140a6d2aad40f223c770e127bceda7cfe17\": container with ID starting with 1dd83886aa3dee176f719060f34d8140a6d2aad40f223c770e127bceda7cfe17 not found: ID does not exist" containerID="1dd83886aa3dee176f719060f34d8140a6d2aad40f223c770e127bceda7cfe17"
Oct 31 03:18:51.118152 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.118122    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd83886aa3dee176f719060f34d8140a6d2aad40f223c770e127bceda7cfe17"} err="failed to get container status \"1dd83886aa3dee176f719060f34d8140a6d2aad40f223c770e127bceda7cfe17\": rpc error: code = NotFound desc = could not find container \"1dd83886aa3dee176f719060f34d8140a6d2aad40f223c770e127bceda7cfe17\": container with ID starting with 1dd83886aa3dee176f719060f34d8140a6d2aad40f223c770e127bceda7cfe17 not found: ID does not exist"
Oct 31 03:18:51.118152 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.118138    4900 scope.go:122] "RemoveContainer" containerID="05ad7cde6fed2c5b32485c1f6f5255cd2173a788bbae64d208a4c279a381d00d"
Oct 31 03:18:51.118388 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:51.118371    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ad7cde6fed2c5b32485c1f6f5255cd2173a788bbae64d208a4c279a381d00d\": container with ID starting with 05ad7cde6fed2c5b32485c1f6f5255cd2173a788bbae64d208a4c279a381d00d not found: ID does not exist" containerID="05ad7cde6fed2c5b32485c1f6f5255cd2173a788bbae64d208a4c279a381d00d"
Oct 31 03:18:51.118425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.118394    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ad7cde6fed2c5b32485c1f6f5255cd2173a788bbae64d208a4c279a381d00d"} err="failed to get container status \"05ad7cde6fed2c5b32485c1f6f5255cd2173a788bbae64d208a4c279a381d00d\": rpc error: code = NotFound desc = could not find container \"05ad7cde6fed2c5b32485c1f6f5255cd2173a788bbae64d208a4c279a381d00d\": container with ID starting with 05ad7cde6fed2c5b32485c1f6f5255cd2173a788bbae64d208a4c279a381d00d not found: ID does not exist"
Oct 31 03:18:51.297313 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.297280    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"]
Oct 31 03:18:51.300937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.300908    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-15000fe0-openshift-default-858b57d65c-rg7vb"]
Oct 31 03:18:51.592278 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.592241    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"]
Oct 31 03:18:51.592643 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.592631    4900 cpu_manager.go:414] "containerMap: removing container" podUID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerName="istio-proxy"
Oct 31 03:18:51.592683 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.592644    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerName="istio-proxy"
Oct 31 03:18:51.592683 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.592660    4900 cpu_manager.go:414] "containerMap: removing container" podUID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerName="istio-proxy"
Oct 31 03:18:51.592683 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.592665    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerName="istio-proxy"
Oct 31 03:18:51.592763 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.592753    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerName="istio-proxy"
Oct 31 03:18:51.592763 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.592762    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="195f8e1b-34bf-41ff-a708-17364e8a42fd" containerName="istio-proxy"
Oct 31 03:18:51.597225 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.597205    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:51.645641 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.645606    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-b59563-serving-cert\""
Oct 31 03:18:51.651954 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.651920    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"]
Oct 31 03:18:51.740934 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.740897    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:51.741111 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.741066    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-metrics-certs\") pod \"maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:51.741231 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.741203    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f16f01c-772c-4277-8519-8b017eceac4b-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:51.741371 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.741299    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4v4n\" (UniqueName: \"kubernetes.io/projected/9f16f01c-772c-4277-8519-8b017eceac4b-kube-api-access-j4v4n\") pod \"maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:51.842447 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.842342    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:51.842626 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.842465    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-metrics-certs\") pod \"maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:51.842626 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:51.842487    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-serving-cert: secret "maas-api-e2e-worker-w2-b59563-serving-cert" not found
Oct 31 03:18:51.842626 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.842536    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f16f01c-772c-4277-8519-8b017eceac4b-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:51.842626 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:51.842558    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-maas-api-tls podName:9f16f01c-772c-4277-8519-8b017eceac4b nodeName:}" failed. No retries permitted until 2026-10-31 03:18:52.342537003 +0000 UTC m=+685.308870325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-maas-api-tls") pod "maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5" (UID: "9f16f01c-772c-4277-8519-8b017eceac4b") : secret "maas-api-e2e-worker-w2-b59563-serving-cert" not found
Oct 31 03:18:51.842840 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.842622    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4v4n\" (UniqueName: \"kubernetes.io/projected/9f16f01c-772c-4277-8519-8b017eceac4b-kube-api-access-j4v4n\") pod \"maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:51.843234 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.843212    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f16f01c-772c-4277-8519-8b017eceac4b-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:51.844998 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.844978    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-metrics-certs\") pod \"maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:51.862626 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.862584    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:51.897907 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:51.897879    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4v4n\" (UniqueName: \"kubernetes.io/projected/9f16f01c-772c-4277-8519-8b017eceac4b-kube-api-access-j4v4n\") pod \"maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:52.075616 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:52.075507    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:52.306872 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:52.306838    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:52.347173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:52.347140    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:52.349537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:52.349515    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-maas-api-tls\") pod \"maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:52.509513 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:52.509478    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:52.783680 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:52.783642    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:52.881853 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:52.881829    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"]
Oct 31 03:18:52.884688 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:18:52.884655    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f16f01c_772c_4277_8519_8b017eceac4b.slice/crio-5d492f579d445e7de1598956065616ddb725d97d34df93c56a7401f89cec1362 WatchSource:0}: Error finding container 5d492f579d445e7de1598956065616ddb725d97d34df93c56a7401f89cec1362: Status 404 returned error can't find the container with id 5d492f579d445e7de1598956065616ddb725d97d34df93c56a7401f89cec1362
Oct 31 03:18:52.886474 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:52.886456    4900 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:18:53.009113 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:53.009085    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:53.111024 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:53.110994    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5" event={"ID":"9f16f01c-772c-4277-8519-8b017eceac4b","Type":"ContainerStarted","Data":"5d492f579d445e7de1598956065616ddb725d97d34df93c56a7401f89cec1362"}
Oct 31 03:18:53.150859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:53.150773    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="195f8e1b-34bf-41ff-a708-17364e8a42fd" path="/var/lib/kubelet/pods/195f8e1b-34bf-41ff-a708-17364e8a42fd/volumes"
Oct 31 03:18:54.116854 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:54.116745    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5" event={"ID":"9f16f01c-772c-4277-8519-8b017eceac4b","Type":"ContainerStarted","Data":"72da6a4818b4d09a5b8ea85b37ed50d66d395aee9da1b3f589424175f0c42e4a"}
Oct 31 03:18:54.117249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:54.116967    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:18:54.225982 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:54.225930    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5" podStartSLOduration=2.276308385 podStartE2EDuration="3.225913921s" totalImagesPullingTime="949.605536ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:54.224376199 +0000 UTC m=+687.190709543" watchObservedRunningTime="2026-10-31 03:18:54.225913921 +0000 UTC m=+687.192247265"
Oct 31 03:18:56.939282 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:56.939242    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"]
Oct 31 03:18:56.942912 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:56.942893    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:56.991866 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:56.991834    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-dd5b85-serving-cert\""
Oct 31 03:18:57.039567 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.039532    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"]
Oct 31 03:18:57.093059 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.093028    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:57.093238 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.093075    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e225e-0ba9-499d-9372-7bd7527354c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:57.093291 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.093232    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-metrics-certs\") pod \"maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:57.093291 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.093274    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lclh\" (UniqueName: \"kubernetes.io/projected/8c7e225e-0ba9-499d-9372-7bd7527354c7-kube-api-access-7lclh\") pod \"maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:57.194720 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.194621    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:57.194720 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.194666    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e225e-0ba9-499d-9372-7bd7527354c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:57.194720 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.194716    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-metrics-certs\") pod \"maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:57.195010 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.194740    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lclh\" (UniqueName: \"kubernetes.io/projected/8c7e225e-0ba9-499d-9372-7bd7527354c7-kube-api-access-7lclh\") pod \"maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:57.195010 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:57.194775    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-serving-cert: secret "maas-api-e2e-worker-w3-dd5b85-serving-cert" not found
Oct 31 03:18:57.195010 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:18:57.194889    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-maas-api-tls podName:8c7e225e-0ba9-499d-9372-7bd7527354c7 nodeName:}" failed. No retries permitted until 2026-10-31 03:18:57.694867994 +0000 UTC m=+690.661201336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-maas-api-tls") pod "maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x" (UID: "8c7e225e-0ba9-499d-9372-7bd7527354c7") : secret "maas-api-e2e-worker-w3-dd5b85-serving-cert" not found
Oct 31 03:18:57.195437 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.195413    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e225e-0ba9-499d-9372-7bd7527354c7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:57.197687 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.197668    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-metrics-certs\") pod \"maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:57.227752 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.227719    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:57.250665 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.250634    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lclh\" (UniqueName: \"kubernetes.io/projected/8c7e225e-0ba9-499d-9372-7bd7527354c7-kube-api-access-7lclh\") pod \"maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:57.448052 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.447961    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:57.665226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.665192    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:18:57.698744 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.698658    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:57.702254 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.702222    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:57.855719 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:57.855688    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:58.223553 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:58.223513    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"]
Oct 31 03:18:58.227489 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:18:58.227463    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c7e225e_0ba9_499d_9372_7bd7527354c7.slice/crio-35d708329f9cc9d21a1b23fa56687a3c97922b9f29c0db1ae653ad0d5c2bd122 WatchSource:0}: Error finding container 35d708329f9cc9d21a1b23fa56687a3c97922b9f29c0db1ae653ad0d5c2bd122: Status 404 returned error can't find the container with id 35d708329f9cc9d21a1b23fa56687a3c97922b9f29c0db1ae653ad0d5c2bd122
Oct 31 03:18:59.153403 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:59.153377    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:18:59.153535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:59.153422    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x" event={"ID":"8c7e225e-0ba9-499d-9372-7bd7527354c7","Type":"ContainerStarted","Data":"f807def13f5fe8e958468f27d9dcebc9293fc593887d726da188537219e6384d"}
Oct 31 03:18:59.153535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:59.153442    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x" event={"ID":"8c7e225e-0ba9-499d-9372-7bd7527354c7","Type":"ContainerStarted","Data":"35d708329f9cc9d21a1b23fa56687a3c97922b9f29c0db1ae653ad0d5c2bd122"}
Oct 31 03:18:59.300846 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:18:59.300733    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x" podStartSLOduration=2.492846766 podStartE2EDuration="3.300719179s" totalImagesPullingTime="807.872413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:18:59.299001529 +0000 UTC m=+692.265334874" watchObservedRunningTime="2026-10-31 03:18:59.300719179 +0000 UTC m=+692.267052523"
Oct 31 03:19:00.087465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:00.087433    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"
Oct 31 03:19:00.126687 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:00.126653    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 03:19:02.082952 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:02.082922    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:02.304157 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:02.304127    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:02.512136 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:02.512106    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:03.720096 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:03.720064    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:03.927953 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:03.927918    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:04.176179 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:04.176140    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:04.385126 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:04.385095    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:04.825233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:04.825173    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:05.164167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:05.164092    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 03:19:05.648779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:05.648747    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:06.075538 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.075504    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"]
Oct 31 03:19:06.079416 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.079389    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.083745 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.083723    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.083892 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.083774    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.083892 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.083857    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.083892 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.083886    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.084064 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.083917    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.084064 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.084021    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.084176 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.084075    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhn96\" (UniqueName: \"kubernetes.io/projected/a63cf118-4a57-48ba-852b-bcaa10da6d4b-kube-api-access-rhn96\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.084176 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.084132    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.084280 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.084209    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.128776 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.128741    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7778d0-openshift-default-dockercfg-c48rb\""
Oct 31 03:19:06.179817 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.179766    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"]
Oct 31 03:19:06.187007 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.186974    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.187255 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.187226    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.187374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.187283    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhn96\" (UniqueName: \"kubernetes.io/projected/a63cf118-4a57-48ba-852b-bcaa10da6d4b-kube-api-access-rhn96\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.187374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.187315    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.187374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.187343    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.187537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.187389    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.187537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.187442    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.187537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.187477    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.187537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.187500    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.187737 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.187701    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.188037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.188014    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.188199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.188175    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.188524 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.188498    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.188754 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.188731    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.192369 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.192348    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.195555 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.195531    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.241676 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.241646    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhn96\" (UniqueName: \"kubernetes.io/projected/a63cf118-4a57-48ba-852b-bcaa10da6d4b-kube-api-access-rhn96\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.241857 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.241642    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.254470 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.254440    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:06.393097 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.393012    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:06.471644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.471610    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:06.808864 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.808841    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"]
Oct 31 03:19:06.810378 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:19:06.810346    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63cf118_4a57_48ba_852b_bcaa10da6d4b.slice/crio-2305ae837a87b35e0c32a7ce4ddfd199933b1b7cb444c4bd9f017437f0250bb8 WatchSource:0}: Error finding container 2305ae837a87b35e0c32a7ce4ddfd199933b1b7cb444c4bd9f017437f0250bb8: Status 404 returned error can't find the container with id 2305ae837a87b35e0c32a7ce4ddfd199933b1b7cb444c4bd9f017437f0250bb8
Oct 31 03:19:06.812662 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.812631    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:19:06.812743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.812691    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:19:06.812743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:06.812718    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:19:06.984355 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:19:06.984316    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:19:07.196212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:07.196115    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4" event={"ID":"a63cf118-4a57-48ba-852b-bcaa10da6d4b","Type":"ContainerStarted","Data":"97b27b953a11ae70ed57bd26c14b0f15067f80ddcf8682cc28289d0dff2e54ae"}
Oct 31 03:19:07.196212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:07.196182    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4" event={"ID":"a63cf118-4a57-48ba-852b-bcaa10da6d4b","Type":"ContainerStarted","Data":"2305ae837a87b35e0c32a7ce4ddfd199933b1b7cb444c4bd9f017437f0250bb8"}
Oct 31 03:19:07.306766 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:07.306694    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4" podStartSLOduration=1.306679677 podStartE2EDuration="1.306679677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:07.306317584 +0000 UTC m=+700.272650927" watchObservedRunningTime="2026-10-31 03:19:07.306679677 +0000 UTC m=+700.273013024"
Oct 31 03:19:07.394121 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:07.394089    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:07.399303 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:07.399275    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:08.201851 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:08.201823    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:08.202663 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:08.202647    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:09.667466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:09.667431    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:10.232637 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:10.232608    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:10.578278 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:10.578176    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:14.358807 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:14.358407    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:14.566886 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:14.566833    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:15.265696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:15.265662    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:15.471965 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:15.471928    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:15.678601 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:15.678508    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:20.993778 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:20.993744    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:21.291041 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:21.290959    4900 generic.go:356] "Generic (PLEG): container finished" podID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerID="98077e4376b1c2336b924aff4b23068ecad4a75e30a153ae0b0650b0453defa5" exitCode=0
Oct 31 03:19:21.291168 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:21.291034    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs" event={"ID":"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8","Type":"ContainerDied","Data":"98077e4376b1c2336b924aff4b23068ecad4a75e30a153ae0b0650b0453defa5"}
Oct 31 03:19:21.291410 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:21.291397    4900 scope.go:122] "RemoveContainer" containerID="98077e4376b1c2336b924aff4b23068ecad4a75e30a153ae0b0650b0453defa5"
Oct 31 03:19:21.292655 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:21.292626    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:19:21.292754 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:21.292728    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:19:21.292841 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:21.292776    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:19:22.012008 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:22.011974    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:19:22.298882 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:22.298774    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs" event={"ID":"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8","Type":"ContainerStarted","Data":"3a4d57284d835ded3e2007e7495c4c1ea25d4c6fe59282accf1db5e98a607c69"}
Oct 31 03:19:23.288174 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:23.288146    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:19:23.289186 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:23.289165    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 03:19:23.289277 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:23.289218    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs" podUID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 03:19:24.288029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:24.287983    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 03:19:24.288298 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:24.288046    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs" podUID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 03:19:25.287903 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:25.287867    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 03:19:25.288084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:25.287915    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs" podUID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 03:19:26.288240 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:26.287763    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 03:19:26.288240 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:26.287864    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs" podUID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 03:19:26.498217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:26.498155    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:27.287443 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:27.287408    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Oct 31 03:19:27.287711 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:27.287684    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs" podUID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Oct 31 03:19:28.264539 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:28.264486    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:28.292000 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:28.291970    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:19:28.292282 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:28.292260    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:19:28.292947 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:28.292932    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:19:28.900129 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:28.900077    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:29.121563 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:29.121528    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:31.973293 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:31.973258    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"]
Oct 31 03:19:31.977264 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:31.977245    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.026818 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.026757    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:32.072486 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.072454    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b59563\"/\"kube-root-ca.crt\""
Oct 31 03:19:32.072486 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.072468    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b59563\"/\"openshift-service-ca.crt\""
Oct 31 03:19:32.072695 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.072546    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b59563\"/\"default-dockercfg-jq9mp\""
Oct 31 03:19:32.074644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.074624    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"]
Oct 31 03:19:32.075434 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.075415    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b59563\"/\"simulator-w2-b59563-kserve-self-signed-certs\""
Oct 31 03:19:32.106952 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.106915    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0f8b95-9b82-406c-8909-f8a11c81a753-tls-certs\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.107114 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.106977    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-home\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.107114 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.107079    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-tmp-dir\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.107233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.107216    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-dshm\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.107304 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.107278    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-model-cache\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.107414 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.107400    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6h4n\" (UniqueName: \"kubernetes.io/projected/8e0f8b95-9b82-406c-8909-f8a11c81a753-kube-api-access-r6h4n\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.208518 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.208486    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-dshm\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.208518 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.208522    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-model-cache\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.208731 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.208588    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6h4n\" (UniqueName: \"kubernetes.io/projected/8e0f8b95-9b82-406c-8909-f8a11c81a753-kube-api-access-r6h4n\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.208731 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.208642    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0f8b95-9b82-406c-8909-f8a11c81a753-tls-certs\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.208731 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.208671    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-home\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.208908 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.208871    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-tmp-dir\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.209072 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.209047    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-model-cache\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.209151 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.209079    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-home\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.209267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.209245    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-tmp-dir\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.210882 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.210854    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-dshm\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.211122 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.211107    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0f8b95-9b82-406c-8909-f8a11c81a753-tls-certs\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.235714 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.235675    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:32.264476 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.264441    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6h4n\" (UniqueName: \"kubernetes.io/projected/8e0f8b95-9b82-406c-8909-f8a11c81a753-kube-api-access-r6h4n\") pod \"simulator-w2-b59563-kserve-c95f846fc-jhgcl\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") " pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.296078 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.296045    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:32.669828 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:32.669782    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"]
Oct 31 03:19:32.672345 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:19:32.672315    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e0f8b95_9b82_406c_8909_f8a11c81a753.slice/crio-37a58c6fdd18769c9d33a1e8110981ebfc5b2fc0e9f2da2840f86b9eedf69b9a WatchSource:0}: Error finding container 37a58c6fdd18769c9d33a1e8110981ebfc5b2fc0e9f2da2840f86b9eedf69b9a: Status 404 returned error can't find the container with id 37a58c6fdd18769c9d33a1e8110981ebfc5b2fc0e9f2da2840f86b9eedf69b9a
Oct 31 03:19:33.353999 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:33.353958    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" event={"ID":"8e0f8b95-9b82-406c-8909-f8a11c81a753","Type":"ContainerStarted","Data":"b10e6a986d3cd85f095dfe6a70da7ed3849a7702550421a96dcc78457774fac2"}
Oct 31 03:19:33.354483 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:33.354011    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" event={"ID":"8e0f8b95-9b82-406c-8909-f8a11c81a753","Type":"ContainerStarted","Data":"37a58c6fdd18769c9d33a1e8110981ebfc5b2fc0e9f2da2840f86b9eedf69b9a"}
Oct 31 03:19:33.355568 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:33.355545    4900 generic.go:356] "Generic (PLEG): container finished" podID="9b030d08-9d78-4c66-8729-243e64177d52" containerID="9b0a944a28df52f75b3d7b0d87d896d03519463561f8948d6f52023db2e66149" exitCode=0
Oct 31 03:19:33.355690 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:33.355620    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" event={"ID":"9b030d08-9d78-4c66-8729-243e64177d52","Type":"ContainerDied","Data":"9b0a944a28df52f75b3d7b0d87d896d03519463561f8948d6f52023db2e66149"}
Oct 31 03:19:33.355986 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:33.355972    4900 scope.go:122] "RemoveContainer" containerID="9b0a944a28df52f75b3d7b0d87d896d03519463561f8948d6f52023db2e66149"
Oct 31 03:19:33.357145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:33.357121    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:19:33.357197 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:33.357184    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:19:33.357231 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:33.357215    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:19:33.461808 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:33.461745    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" podStartSLOduration=2.461730142 podStartE2EDuration="2.461730142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:33.460346418 +0000 UTC m=+726.426679763" watchObservedRunningTime="2026-10-31 03:19:33.461730142 +0000 UTC m=+726.428063521"
Oct 31 03:19:34.080686 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:34.080650    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:19:34.296574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:34.296539    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:34.367014 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:34.366920    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" event={"ID":"9b030d08-9d78-4c66-8729-243e64177d52","Type":"ContainerStarted","Data":"231e5d0ad082c4ac5e352726fd6503fd897abe7f680c522c6929df488597c96d"}
Oct 31 03:19:34.913860 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:34.913826    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:19:34.914854 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:34.914831    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 03:19:34.914925 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:34.914880    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" podUID="9b030d08-9d78-4c66-8729-243e64177d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 03:19:35.092194 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:35.092164    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:35.707387 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:35.707349    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:35.913920 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:35.913880    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 03:19:35.914089 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:35.913939    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" podUID="9b030d08-9d78-4c66-8729-243e64177d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 03:19:35.945811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:35.941317    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:36.913607 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:36.913568    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 03:19:36.914032 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:36.913626    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" podUID="9b030d08-9d78-4c66-8729-243e64177d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 03:19:36.985139 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:19:36.985098    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:19:37.169600 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.169527    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"]
Oct 31 03:19:37.172523 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.172508    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:37.220418 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.220390    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"]
Oct 31 03:19:37.265072 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.265036    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-7778d0-serving-cert\""
Oct 31 03:19:37.284110 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.284080    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkp55\" (UniqueName: \"kubernetes.io/projected/0082baa4-f377-4f35-ac76-961aa2045b86-kube-api-access-vkp55\") pod \"maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:37.284271 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.284194    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-metrics-certs\") pod \"maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:37.284271 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.284259    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:37.284402 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.284379    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0082baa4-f377-4f35-ac76-961aa2045b86-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:37.385535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.385506    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkp55\" (UniqueName: \"kubernetes.io/projected/0082baa4-f377-4f35-ac76-961aa2045b86-kube-api-access-vkp55\") pod \"maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:37.385701 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.385574    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-metrics-certs\") pod \"maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:37.385742 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.385726    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:37.385868 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.385847    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0082baa4-f377-4f35-ac76-961aa2045b86-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:37.385927 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:19:37.385866    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-serving-cert: secret "maas-api-e2e-worker-w0-7778d0-serving-cert" not found
Oct 31 03:19:37.385971 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:19:37.385939    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-maas-api-tls podName:0082baa4-f377-4f35-ac76-961aa2045b86 nodeName:}" failed. No retries permitted until 2026-10-31 03:19:37.885923398 +0000 UTC m=+730.852256721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-maas-api-tls") pod "maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt" (UID: "0082baa4-f377-4f35-ac76-961aa2045b86") : secret "maas-api-e2e-worker-w0-7778d0-serving-cert" not found
Oct 31 03:19:37.386487 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.386470    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0082baa4-f377-4f35-ac76-961aa2045b86-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:37.387937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.387919    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-metrics-certs\") pod \"maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:37.460159 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.460059    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:37.462954 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.462929    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkp55\" (UniqueName: \"kubernetes.io/projected/0082baa4-f377-4f35-ac76-961aa2045b86-kube-api-access-vkp55\") pod \"maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:37.697076 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.697021    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:37.891399 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.891357    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:37.893855 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.893829    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-maas-api-tls\") pod \"maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:37.913737 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.913706    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 03:19:37.914079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:37.913751    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" podUID="9b030d08-9d78-4c66-8729-243e64177d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 03:19:38.084780 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:38.084748    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:38.470529 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:38.470501    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"]
Oct 31 03:19:38.472664 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:19:38.472633    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0082baa4_f377_4f35_ac76_961aa2045b86.slice/crio-ea73ff0a7a3cda677f353bcc5ff6fa20466aa99920a06cc5c892f52fe5781061 WatchSource:0}: Error finding container ea73ff0a7a3cda677f353bcc5ff6fa20466aa99920a06cc5c892f52fe5781061: Status 404 returned error can't find the container with id ea73ff0a7a3cda677f353bcc5ff6fa20466aa99920a06cc5c892f52fe5781061
Oct 31 03:19:38.913907 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:38.913867    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 03:19:38.914362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:38.913920    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" podUID="9b030d08-9d78-4c66-8729-243e64177d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 03:19:39.403482 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:39.403422    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt" event={"ID":"0082baa4-f377-4f35-ac76-961aa2045b86","Type":"ContainerStarted","Data":"ea73ff0a7a3cda677f353bcc5ff6fa20466aa99920a06cc5c892f52fe5781061"}
Oct 31 03:19:39.917332 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:39.917300    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:19:39.917752 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:39.917593    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:19:39.918370 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:39.918353    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:19:40.409765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:40.409728    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt" event={"ID":"0082baa4-f377-4f35-ac76-961aa2045b86","Type":"ContainerStarted","Data":"12eb4b66e0278f5fad79ee1ad49d9e3aba4706145ac489155e63f14ee404c331"}
Oct 31 03:19:40.521328 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:40.521269    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt" podStartSLOduration=2.6556055880000002 podStartE2EDuration="3.521255034s" totalImagesPullingTime="865.649446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:19:40.519032699 +0000 UTC m=+733.485366043" watchObservedRunningTime="2026-10-31 03:19:40.521255034 +0000 UTC m=+733.487588378"
Oct 31 03:19:41.415072 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:41.415038    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:42.186511 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:42.186467    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:42.297163 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:42.297123    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:42.309729 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:42.309702    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:42.431083 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:42.431047    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:19:42.768300 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:42.768261    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:42.982022 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:42.981994    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:43.198533 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:43.198442    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:45.677760 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:45.677724    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:47.426169 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:47.426138    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 03:19:48.006183 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:48.006148    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:52.872097 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:52.872038    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:53.081255 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:53.081222    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:54.244033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:54.243977    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:19:54.244586 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:54.244046    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4" podUID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:19:55.505259 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:55.505225    4900 generic.go:356] "Generic (PLEG): container finished" podID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerID="97b27b953a11ae70ed57bd26c14b0f15067f80ddcf8682cc28289d0dff2e54ae" exitCode=0
Oct 31 03:19:55.505822 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:55.505298    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4" event={"ID":"a63cf118-4a57-48ba-852b-bcaa10da6d4b","Type":"ContainerDied","Data":"97b27b953a11ae70ed57bd26c14b0f15067f80ddcf8682cc28289d0dff2e54ae"}
Oct 31 03:19:55.505917 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:55.505823    4900 scope.go:122] "RemoveContainer" containerID="97b27b953a11ae70ed57bd26c14b0f15067f80ddcf8682cc28289d0dff2e54ae"
Oct 31 03:19:55.507125 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:55.507093    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:19:55.507216 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:55.507159    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:19:55.507216 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:55.507189    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:19:56.513040 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:56.512998    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4" event={"ID":"a63cf118-4a57-48ba-852b-bcaa10da6d4b","Type":"ContainerStarted","Data":"ba93d0fd0f3edcfc90bc1e8d597e76e466d068f3041b752c6bf307d90018cfd9"}
Oct 31 03:19:57.099782 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:57.099752    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:57.393738 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:57.393647    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:19:57.394002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:57.393980    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 31 03:19:57.394051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:57.394027    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4" podUID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 31 03:19:57.732175 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:57.732147    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:57.953963 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:57.953928    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:58.169428 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:58.169336    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:19:58.394071 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:58.394031    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 31 03:19:58.394249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:58.394103    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4" podUID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 31 03:19:59.393999 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:59.393961    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 31 03:19:59.394420 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:19:59.394029    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4" podUID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 31 03:20:00.394065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:00.394031    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 31 03:20:00.394469 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:00.394093    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4" podUID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 31 03:20:01.393594 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.393562    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Oct 31 03:20:01.393772 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.393615    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4" podUID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Oct 31 03:20:01.447237 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.447187    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"]
Oct 31 03:20:01.451487 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.451463    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.499544 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.499511    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"]
Oct 31 03:20:01.543980 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.543928    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9360eb\"/\"openshift-service-ca.crt\""
Oct 31 03:20:01.543980 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.543963    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9360eb\"/\"kube-root-ca.crt\""
Oct 31 03:20:01.544251 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.544014    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9360eb\"/\"default-dockercfg-w6g9t\""
Oct 31 03:20:01.546046 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.546024    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9360eb\"/\"simulator-w4-9360eb-kserve-self-signed-certs\""
Oct 31 03:20:01.572769 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.572724    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-home\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.573010 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.572821    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-dshm\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.573010 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.572919    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-model-cache\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.573010 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.572943    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-tmp-dir\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.573010 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.572976    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0371044e-9bea-4b59-8329-2aa223d2c126-tls-certs\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.573253 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.573086    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc8ng\" (UniqueName: \"kubernetes.io/projected/0371044e-9bea-4b59-8329-2aa223d2c126-kube-api-access-lc8ng\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.607381 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.607341    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:01.674004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.673903    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0371044e-9bea-4b59-8329-2aa223d2c126-tls-certs\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.674004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.673951    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc8ng\" (UniqueName: \"kubernetes.io/projected/0371044e-9bea-4b59-8329-2aa223d2c126-kube-api-access-lc8ng\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.674004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.673998    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-home\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.674303 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.674054    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-dshm\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.674303 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.674139    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-model-cache\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.674303 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.674170    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-tmp-dir\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.674536 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.674509    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-home\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.674605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.674549    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-tmp-dir\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.674605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.674590    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-model-cache\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.676429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.676403    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-dshm\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.676782 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.676760    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0371044e-9bea-4b59-8329-2aa223d2c126-tls-certs\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.727255 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.727214    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc8ng\" (UniqueName: \"kubernetes.io/projected/0371044e-9bea-4b59-8329-2aa223d2c126-kube-api-access-lc8ng\") pod \"simulator-w4-9360eb-kserve-7449fb8744-ff5cz\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") " pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:01.770126 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:01.770091    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:02.136147 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:02.136122    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"]
Oct 31 03:20:02.138680 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:20:02.138650    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0371044e_9bea_4b59_8329_2aa223d2c126.slice/crio-3c6cf126bb1dc8fb623e19687544df5aedf91ee47c774c62455ac56da25fea18 WatchSource:0}: Error finding container 3c6cf126bb1dc8fb623e19687544df5aedf91ee47c774c62455ac56da25fea18: Status 404 returned error can't find the container with id 3c6cf126bb1dc8fb623e19687544df5aedf91ee47c774c62455ac56da25fea18
Oct 31 03:20:02.397515 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:02.397434    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:20:02.397748 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:02.397728    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:20:02.398578 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:02.398561    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:20:02.569075 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:02.569030    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" event={"ID":"0371044e-9bea-4b59-8329-2aa223d2c126","Type":"ContainerStarted","Data":"7a66eaa92636a09d8de4d93d48f6844816dfd4a3c9e42273083ac3c4484b42ea"}
Oct 31 03:20:02.569513 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:02.569082    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" event={"ID":"0371044e-9bea-4b59-8329-2aa223d2c126","Type":"ContainerStarted","Data":"3c6cf126bb1dc8fb623e19687544df5aedf91ee47c774c62455ac56da25fea18"}
Oct 31 03:20:02.729092 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:02.728973    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" podStartSLOduration=1.728957282 podStartE2EDuration="1.728957282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:02.727371093 +0000 UTC m=+755.693704431" watchObservedRunningTime="2026-10-31 03:20:02.728957282 +0000 UTC m=+755.695290671"
Oct 31 03:20:02.770812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:02.770743    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:03.059378 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:03.059345    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:06.986088 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:20:06.986048    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:20:08.110912 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:08.110878    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:11.771193 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:11.771157    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:11.783827 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:11.783777    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:12.628268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:12.628234    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:20:12.993108 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:12.993068    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:13.729162 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:13.729131    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:16.296254 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:16.296218    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:16.873337 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:16.873297    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:17.095274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:17.095222    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:18.274160 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:18.274119    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:23.190817 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:23.190763    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:28.103347 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:28.103310    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:32.986166 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:32.986133    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:33.536161 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:33.536131    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:33.745164 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:33.745131    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:34.820226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:34.820192    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"]
Oct 31 03:20:34.824055 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:34.824034    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:34.877807 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:34.877751    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"]
Oct 31 03:20:34.920896 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:34.920861    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b59563\"/\"premium-w2-b59563-kserve-self-signed-certs\""
Oct 31 03:20:34.940758 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:34.940719    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-home\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:34.940758 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:34.940761    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8s5d\" (UniqueName: \"kubernetes.io/projected/397fbdbf-8683-4d96-aa94-5e8ea77a785d-kube-api-access-s8s5d\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:34.940998 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:34.940927    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/397fbdbf-8683-4d96-aa94-5e8ea77a785d-tls-certs\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:34.940998 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:34.940968    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-model-cache\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:34.941076 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:34.941038    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-tmp-dir\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:34.941076 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:34.941063    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-dshm\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:34.981562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:34.981522    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:35.041671 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.041636    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/397fbdbf-8683-4d96-aa94-5e8ea77a785d-tls-certs\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:35.041880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.041685    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-model-cache\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:35.041880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.041740    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-tmp-dir\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:35.041880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.041767    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-dshm\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:35.041880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.041872    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-home\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:35.042092 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.041904    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8s5d\" (UniqueName: \"kubernetes.io/projected/397fbdbf-8683-4d96-aa94-5e8ea77a785d-kube-api-access-s8s5d\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:35.042276 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.042237    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-tmp-dir\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:35.042405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.042250    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-home\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:35.042405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.042339    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-model-cache\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:35.044131 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.044104    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-dshm\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:35.044358 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.044340    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/397fbdbf-8683-4d96-aa94-5e8ea77a785d-tls-certs\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:35.095044 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.094960    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8s5d\" (UniqueName: \"kubernetes.io/projected/397fbdbf-8683-4d96-aa94-5e8ea77a785d-kube-api-access-s8s5d\") pod \"premium-w2-b59563-kserve-74856ddf84-hh7th\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") " pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:35.135336 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.135294    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:35.511938 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.511910    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"]
Oct 31 03:20:35.513324 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:20:35.513299    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod397fbdbf_8683_4d96_aa94_5e8ea77a785d.slice/crio-c560be3d64d8423b955de775f952222f9e22af667eaa86dfd9c9a3d499f41dcd WatchSource:0}: Error finding container c560be3d64d8423b955de775f952222f9e22af667eaa86dfd9c9a3d499f41dcd: Status 404 returned error can't find the container with id c560be3d64d8423b955de775f952222f9e22af667eaa86dfd9c9a3d499f41dcd
Oct 31 03:20:35.731509 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.731475    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" event={"ID":"397fbdbf-8683-4d96-aa94-5e8ea77a785d","Type":"ContainerStarted","Data":"be3400c115b8296d0004971f09224dcb890b1d20dda4d158c94ffad189ea3460"}
Oct 31 03:20:35.731683 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.731519    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" event={"ID":"397fbdbf-8683-4d96-aa94-5e8ea77a785d","Type":"ContainerStarted","Data":"c560be3d64d8423b955de775f952222f9e22af667eaa86dfd9c9a3d499f41dcd"}
Oct 31 03:20:35.839607 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:35.839503    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" podStartSLOduration=1.839487733 podStartE2EDuration="1.839487733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:35.837545748 +0000 UTC m=+788.803879092" watchObservedRunningTime="2026-10-31 03:20:35.839487733 +0000 UTC m=+788.805821077"
Oct 31 03:20:36.136023 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:36.135930    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:36.984530 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:20:36.984492    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:20:37.875055 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:37.875023    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:42.904363 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:42.904320    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:45.136449 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:45.136415    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:45.152379 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:45.152352    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:45.798737 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:45.798700    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:45.799908 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:45.799887    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:20:46.011374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:46.011344    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:48.005121 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:48.005086    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:52.765320 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.765285    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"]
Oct 31 03:20:52.768937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.768916    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.825505 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.825473    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-home\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.825678 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.825516    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-tmp-dir\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.825678 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.825587    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9q5p\" (UniqueName: \"kubernetes.io/projected/e58cb630-6755-44c8-b068-2bbdb6d4fe15-kube-api-access-s9q5p\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.825824 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.825749    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-model-cache\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.825824 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.825779    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e58cb630-6755-44c8-b068-2bbdb6d4fe15-tls-certs\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.825991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.825966    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-dshm\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.866002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.865964    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7778d0\"/\"openshift-service-ca.crt\""
Oct 31 03:20:52.866002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.865992    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7778d0\"/\"kube-root-ca.crt\""
Oct 31 03:20:52.866206 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.866080    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7778d0\"/\"simulator-w0-7778d0-kserve-self-signed-certs\""
Oct 31 03:20:52.866206 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.866091    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7778d0\"/\"default-dockercfg-7dmx6\""
Oct 31 03:20:52.868229 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.868207    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"]
Oct 31 03:20:52.927533 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.927499    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-dshm\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.927694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.927591    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-home\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.927694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.927623    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-tmp-dir\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.927694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.927669    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9q5p\" (UniqueName: \"kubernetes.io/projected/e58cb630-6755-44c8-b068-2bbdb6d4fe15-kube-api-access-s9q5p\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.927893 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.927737    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-model-cache\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.927893 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.927768    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e58cb630-6755-44c8-b068-2bbdb6d4fe15-tls-certs\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.928072 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.928051    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-tmp-dir\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.928150 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.928087    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-home\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.928195 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.928149    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-model-cache\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.929827 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.929806    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-dshm\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.930241 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.930225    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e58cb630-6755-44c8-b068-2bbdb6d4fe15-tls-certs\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:52.935195 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.935165    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:52.981277 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:52.981249    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9q5p\" (UniqueName: \"kubernetes.io/projected/e58cb630-6755-44c8-b068-2bbdb6d4fe15-kube-api-access-s9q5p\") pod \"simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") " pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:53.081029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:53.080943    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:53.148215 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:53.146716    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:20:53.456224 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:53.456201    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"]
Oct 31 03:20:53.458079 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:20:53.458057    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode58cb630_6755_44c8_b068_2bbdb6d4fe15.slice/crio-b3d4bce1d00a0567f2d176e5ec5fbbdc4e24796ec80f902d15104ac742688992 WatchSource:0}: Error finding container b3d4bce1d00a0567f2d176e5ec5fbbdc4e24796ec80f902d15104ac742688992: Status 404 returned error can't find the container with id b3d4bce1d00a0567f2d176e5ec5fbbdc4e24796ec80f902d15104ac742688992
Oct 31 03:20:53.824999 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:53.824967    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp" event={"ID":"e58cb630-6755-44c8-b068-2bbdb6d4fe15","Type":"ContainerStarted","Data":"8c72a998c6d8a3e085801d5a2677f767073304a31cc8684e55fda64dfba0b9e9"}
Oct 31 03:20:53.824999 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:53.825007    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp" event={"ID":"e58cb630-6755-44c8-b068-2bbdb6d4fe15","Type":"ContainerStarted","Data":"b3d4bce1d00a0567f2d176e5ec5fbbdc4e24796ec80f902d15104ac742688992"}
Oct 31 03:20:53.936895 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:53.936817    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp" podStartSLOduration=1.936778718 podStartE2EDuration="1.936778718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:53.93539389 +0000 UTC m=+806.901727234" watchObservedRunningTime="2026-10-31 03:20:53.936778718 +0000 UTC m=+806.903112062"
Oct 31 03:20:54.081090 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:54.081005    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:20:57.985257 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:20:57.985222    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:03.016926 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:03.016890    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:03.081620 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:03.081592    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:21:03.093730 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:03.093705    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:21:03.885419 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:03.885390    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:21:06.428549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:06.428514    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:06.985097 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:21:06.985061    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:21:07.997002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:07.996968    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:12.884422 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:12.884338    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:17.896805 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:17.896764    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:22.785642 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:22.785610    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:25.448234 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:25.448201    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:27.830624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:27.830590    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:28.925922 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:28.925891    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"]
Oct 31 03:21:28.929421 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:28.929401    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.012712 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.012681    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.012887 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.012721    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.012887 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.012822    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.012887 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.012861    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.012999 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.012928    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.012999 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.012984    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.013064 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.013009    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.013064 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.013033    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.013125 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.013105    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvvj2\" (UniqueName: \"kubernetes.io/projected/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-kube-api-access-kvvj2\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.024551 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.024528    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9360eb-openshift-default-dockercfg-lsbz7\""
Oct 31 03:21:29.026985 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.026960    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"]
Oct 31 03:21:29.113809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.113753    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.113809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.113812    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.114067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.113838    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.114067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.113872    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.114067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.113985    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.114067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.114033    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.114278 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.114086    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvvj2\" (UniqueName: \"kubernetes.io/projected/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-kube-api-access-kvvj2\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.114278 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.114170    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.114278 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.114208    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.114431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.114281    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.114431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.114339    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.114545 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.114522    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.114736 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.114715    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.114885 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.114866    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.116643 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.116620    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.116985 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.116964    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.167243 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.167214    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.167392 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.167332    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvvj2\" (UniqueName: \"kubernetes.io/projected/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-kube-api-access-kvvj2\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.242496 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.242465    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:29.656269 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.656192    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"]
Oct 31 03:21:29.691759 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:21:29.691729    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda1fa9b_5f39_44cb_9f1d_8d64876b7dbe.slice/crio-c7fdf1d9faecf1b680b48bcd2e98090ee237ef204dbcd1d4bcc9719ba9865305 WatchSource:0}: Error finding container c7fdf1d9faecf1b680b48bcd2e98090ee237ef204dbcd1d4bcc9719ba9865305: Status 404 returned error can't find the container with id c7fdf1d9faecf1b680b48bcd2e98090ee237ef204dbcd1d4bcc9719ba9865305
Oct 31 03:21:29.694183 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.694152    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:21:29.694265 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.694215    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:21:29.694265 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:29.694245    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:21:30.003294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:30.003259    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr" event={"ID":"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe","Type":"ContainerStarted","Data":"6e1eadb67b60968a148ceb76ed995fe3146a606d00dd0599eac18910e7a0d1c1"}
Oct 31 03:21:30.003739 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:30.003310    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr" event={"ID":"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe","Type":"ContainerStarted","Data":"c7fdf1d9faecf1b680b48bcd2e98090ee237ef204dbcd1d4bcc9719ba9865305"}
Oct 31 03:21:30.114136 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:30.114080    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr" podStartSLOduration=2.114064975 podStartE2EDuration="2.114064975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:30.111057963 +0000 UTC m=+843.077391310" watchObservedRunningTime="2026-10-31 03:21:30.114064975 +0000 UTC m=+843.080398322"
Oct 31 03:21:30.257874 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:30.256855    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:30.258935 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:30.258901    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:21:30.259029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:30.258968    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr" podUID="bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 31 03:21:31.257812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:31.257158    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:21:31.257812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:31.257222    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr" podUID="bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 31 03:21:32.253812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:32.249157    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:21:32.253812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:32.249234    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr" podUID="bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 31 03:21:32.691134 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:32.691029    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:33.244134 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:33.244098    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:21:33.244420 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:33.244389    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr" podUID="bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 31 03:21:34.255158 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.255122    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:21:34.256463 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.256432    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr" podUID="bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 31 03:21:34.659614 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.659521    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"]
Oct 31 03:21:34.664216 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.664196    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.714612 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.714571    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b59563-openshift-default-dockercfg-h4q7c\""
Oct 31 03:21:34.766231 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.766191    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"]
Oct 31 03:21:34.793593 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.793557    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.793756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.793616    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.793756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.793722    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.793756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.793749    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcxh7\" (UniqueName: \"kubernetes.io/projected/7c853068-dbe4-42a1-83d8-571b1b2b1719-kube-api-access-qcxh7\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.793879 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.793776    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.793910 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.793870    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.793910 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.793896    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c853068-dbe4-42a1-83d8-571b1b2b1719-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.793971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.793914    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.793971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.793960    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.895068 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.895034    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.895234 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.895108    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.895234 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.895146    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.895234 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.895186    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.895234 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.895201    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcxh7\" (UniqueName: \"kubernetes.io/projected/7c853068-dbe4-42a1-83d8-571b1b2b1719-kube-api-access-qcxh7\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.895437 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.895234    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.895437 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.895263    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.895437 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.895292    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c853068-dbe4-42a1-83d8-571b1b2b1719-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.895437 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.895320    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.895662 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.895583    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.895760 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.895724    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.895870 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.895807    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.895870 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.895842    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.896038 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.896021    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c853068-dbe4-42a1-83d8-571b1b2b1719-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.897431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.897411    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.897673 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.897646    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.960498 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.960417    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.960713 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.960692    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcxh7\" (UniqueName: \"kubernetes.io/projected/7c853068-dbe4-42a1-83d8-571b1b2b1719-kube-api-access-qcxh7\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:34.977528 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:34.977505    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:35.258841 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:35.245194    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:21:35.258841 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:35.245252    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr" podUID="bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 31 03:21:35.388553 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:35.388528    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"]
Oct 31 03:21:35.390518 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:21:35.390490    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c853068_dbe4_42a1_83d8_571b1b2b1719.slice/crio-a324af98f5378fe1d4b108243e2054ad11f5f8a0a8542fe0710ec6c1dae8c7f1 WatchSource:0}: Error finding container a324af98f5378fe1d4b108243e2054ad11f5f8a0a8542fe0710ec6c1dae8c7f1: Status 404 returned error can't find the container with id a324af98f5378fe1d4b108243e2054ad11f5f8a0a8542fe0710ec6c1dae8c7f1
Oct 31 03:21:35.392805 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:35.392762    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:21:35.392878 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:35.392838    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:21:35.392878 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:35.392867    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:21:36.055813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:36.054719    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55" event={"ID":"7c853068-dbe4-42a1-83d8-571b1b2b1719","Type":"ContainerStarted","Data":"9545efefdde531249cb84e0abda6048220c8f8ff20ee7dff1088f563c408cfd7"}
Oct 31 03:21:36.055813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:36.054802    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55" event={"ID":"7c853068-dbe4-42a1-83d8-571b1b2b1719","Type":"ContainerStarted","Data":"a324af98f5378fe1d4b108243e2054ad11f5f8a0a8542fe0710ec6c1dae8c7f1"}
Oct 31 03:21:36.470112 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:36.470009    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:36.470778 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:36.470744    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:36.471998 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:36.471979    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 03:21:36.582149 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:36.582095    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55" podStartSLOduration=2.582081603 podStartE2EDuration="2.582081603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:36.171162325 +0000 UTC m=+849.137495670" watchObservedRunningTime="2026-10-31 03:21:36.582081603 +0000 UTC m=+849.548414947"
Oct 31 03:21:36.978110 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:36.978070    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:36.979643 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:36.979610    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:21:36.979767 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:36.979665    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55" podUID="7c853068-dbe4-42a1-83d8-571b1b2b1719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 03:21:36.985330 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:21:36.985305    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:21:37.581140 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:37.581100    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:37.978781 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:37.978689    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:21:37.978781 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:37.978745    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55" podUID="7c853068-dbe4-42a1-83d8-571b1b2b1719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 03:21:38.978463 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:38.978419    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:21:38.978955 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:38.978487    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55" podUID="7c853068-dbe4-42a1-83d8-571b1b2b1719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 03:21:39.978660 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:39.978624    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:21:39.979107 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:39.978678    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55" podUID="7c853068-dbe4-42a1-83d8-571b1b2b1719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 03:21:40.978272 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:40.978236    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:21:40.978447 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:40.978300    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55" podUID="7c853068-dbe4-42a1-83d8-571b1b2b1719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Oct 31 03:21:42.050087 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:42.050054    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:42.050552 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:42.050463    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:42.051284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:42.051260    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:21:42.474157 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:42.474071    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:46.279582 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:46.279547    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:47.694850 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:47.694816    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:52.407432 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:52.407311    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:52.635975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:52.635941    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:21:57.435975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:21:57.435943    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:02.407003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:02.406969    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:06.984646 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:22:06.984605    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:22:07.306091 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:07.306014    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:12.264948 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:12.264915    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:17.399079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:17.399041    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:17.979236 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:17.979195    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:18.250426 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:18.250390    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:22.293767 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:22.293727    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:27.231112 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:27.231077    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:22:27.239135 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:27.239109    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:22:27.305303 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:27.305276    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:32.335061 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:32.335017    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:36.984680 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:22:36.984640    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:22:37.209925 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:37.209889    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:41.734768 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:41.734689    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:41.947612 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:41.947561    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:42.161725 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:42.161643    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:42.386066 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:42.386035    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:46.126167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:46.126132    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:46.698884 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:46.698852    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:47.290242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:47.290212    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:52.326378 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:52.326262    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:22:57.195646 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:22:57.195610    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:00.617417 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:00.617386    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:02.176570 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:02.176519    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:06.984415 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:23:06.984380    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:23:07.216809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:07.216765    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:08.214728 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.214694    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"]
Oct 31 03:23:08.219646 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.219618    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.325279 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.325246    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"]
Oct 31 03:23:08.387331 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.387290    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.390842 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.390760    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaea4507-660f-4b97-b8a7-d349c4441251-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.391075 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.391049    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.391203 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.391094    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v92tl\" (UniqueName: \"kubernetes.io/projected/eaea4507-660f-4b97-b8a7-d349c4441251-kube-api-access-v92tl\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.391203 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.391141    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.391203 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.391183    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.391383 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.391257    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.391383 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.391286    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaea4507-660f-4b97-b8a7-d349c4441251-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.391383 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.391330    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaea4507-660f-4b97-b8a7-d349c4441251-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.491942 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.491908    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.491942 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.491948    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaea4507-660f-4b97-b8a7-d349c4441251-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.492202 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.491967    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaea4507-660f-4b97-b8a7-d349c4441251-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.492202 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.492014    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.492202 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.492125    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaea4507-660f-4b97-b8a7-d349c4441251-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.492372 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.492270    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.492372 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.492327    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v92tl\" (UniqueName: \"kubernetes.io/projected/eaea4507-660f-4b97-b8a7-d349c4441251-kube-api-access-v92tl\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.492481 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.492393    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.492481 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.492436    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.492581 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.492485    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.492581 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.492392    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.492711 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.492683    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.492848 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.492761    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaea4507-660f-4b97-b8a7-d349c4441251-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.492915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.492845    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.494462 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.494441    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.494597 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.494579    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaea4507-660f-4b97-b8a7-d349c4441251-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.545937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.545904    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaea4507-660f-4b97-b8a7-d349c4441251-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.546090 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.546071    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v92tl\" (UniqueName: \"kubernetes.io/projected/eaea4507-660f-4b97-b8a7-d349c4441251-kube-api-access-v92tl\") pod \"maas-default-gateway-openshift-default-6886b96f9-pzd8p\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:08.833212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:08.833118    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:09.255586 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:09.255559    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"]
Oct 31 03:23:09.257416 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:23:09.257383    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaea4507_660f_4b97_b8a7_d349c4441251.slice/crio-600ee7c0a4b781c8558f52a3407379d0b29e718a701d7f477922e9cd7915bebe WatchSource:0}: Error finding container 600ee7c0a4b781c8558f52a3407379d0b29e718a701d7f477922e9cd7915bebe: Status 404 returned error can't find the container with id 600ee7c0a4b781c8558f52a3407379d0b29e718a701d7f477922e9cd7915bebe
Oct 31 03:23:09.259774 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:09.259735    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:23:09.259868 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:09.259843    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:23:09.259908 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:09.259876    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:23:09.586632 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:09.586545    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p" event={"ID":"eaea4507-660f-4b97-b8a7-d349c4441251","Type":"ContainerStarted","Data":"476e26dc39e6c9f18225dd6f0bc00c7ea8a92f14d9fb55a433585ffbb1e96912"}
Oct 31 03:23:09.586632 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:09.586608    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p" event={"ID":"eaea4507-660f-4b97-b8a7-d349c4441251","Type":"ContainerStarted","Data":"600ee7c0a4b781c8558f52a3407379d0b29e718a701d7f477922e9cd7915bebe"}
Oct 31 03:23:09.699158 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:09.699102    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p" podStartSLOduration=1.6990856829999998 podStartE2EDuration="1.699085683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:09.696730545 +0000 UTC m=+942.663063901" watchObservedRunningTime="2026-10-31 03:23:09.699085683 +0000 UTC m=+942.665419028"
Oct 31 03:23:09.834260 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:09.834224    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:10.851823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:10.848918    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pzd8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:23:10.851823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:10.848987    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p" podUID="eaea4507-660f-4b97-b8a7-d349c4441251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 03:23:11.858923 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:11.855925    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pzd8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:23:11.859358 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:11.858971    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p" podUID="eaea4507-660f-4b97-b8a7-d349c4441251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 03:23:12.457279 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:12.457224    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:12.859226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:12.859174    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pzd8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:23:12.859625 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:12.859254    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p" podUID="eaea4507-660f-4b97-b8a7-d349c4441251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 03:23:13.834236 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:13.834195    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pzd8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:23:13.834552 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:13.834520    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p" podUID="eaea4507-660f-4b97-b8a7-d349c4441251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 03:23:14.833843 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:14.833772    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pzd8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:23:14.834328 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:14.833879    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p" podUID="eaea4507-660f-4b97-b8a7-d349c4441251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 03:23:15.834506 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:15.834449    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pzd8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:23:15.834985 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:15.834509    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p" podUID="eaea4507-660f-4b97-b8a7-d349c4441251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 03:23:16.833821 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:16.833762    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pzd8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:23:16.834031 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:16.833850    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p" podUID="eaea4507-660f-4b97-b8a7-d349c4441251" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 03:23:17.810426 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:17.810323    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:18.567288 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:18.567251    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:18.567836 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:18.567800    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:18.568699 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:18.568675    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:23:22.924167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:22.924134    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:28.118570 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:28.118540    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:33.168613 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:33.168570    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:33.856046 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:33.856005    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"]
Oct 31 03:23:33.860845 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:33.860817    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:33.953447 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:33.953409    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"]
Oct 31 03:23:34.029746 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.029693    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.029946 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.029875    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.029946 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.029921    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.030049 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.030014    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.030125 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.030102    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.030335 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.030312    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m46b\" (UniqueName: \"kubernetes.io/projected/0dce71a3-b4c7-459d-92e8-c0515341abf1-kube-api-access-7m46b\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.030425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.030391    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dce71a3-b4c7-459d-92e8-c0515341abf1-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.030515 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.030484    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.030615 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.030563    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.133667 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.133553    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.133667 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.133642    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.133992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.133673    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.133992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.133705    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.133992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.133766    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.133992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.133814    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m46b\" (UniqueName: \"kubernetes.io/projected/0dce71a3-b4c7-459d-92e8-c0515341abf1-kube-api-access-7m46b\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.133992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.133890    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dce71a3-b4c7-459d-92e8-c0515341abf1-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.133992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.133955    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.133992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.133985    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.134660 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.134630    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.135320 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.135291    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.155328 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.154960    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.177351 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.170621    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dce71a3-b4c7-459d-92e8-c0515341abf1-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.177351 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.170978    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.177351 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.171579    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.218532 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.217014    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.218532 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.218326    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.286967 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.286929    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m46b\" (UniqueName: \"kubernetes.io/projected/0dce71a3-b4c7-459d-92e8-c0515341abf1-kube-api-access-7m46b\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:34.479346 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:34.478999    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:35.005816 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:35.000127    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"]
Oct 31 03:23:35.009611 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:35.009069    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:23:35.009611 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:35.009163    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:23:35.009611 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:35.009210    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:23:35.228943 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:35.228894    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pflrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:23:35.229367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:35.228954    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" podUID="baed435c-cff8-4c33-b642-a1fd39d437f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:23:35.751927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:35.751888    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx" event={"ID":"0dce71a3-b4c7-459d-92e8-c0515341abf1","Type":"ContainerStarted","Data":"ddf4554de2aadfe45045269273bff65d8ce203aac0bf23a945ca3f8932676eda"}
Oct 31 03:23:35.752214 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:35.752192    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx" event={"ID":"0dce71a3-b4c7-459d-92e8-c0515341abf1","Type":"ContainerStarted","Data":"a61565008bb1996ccb098faad7f38695b8e8805dcf623c4e6fa94b11911bd1f3"}
Oct 31 03:23:35.964967 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:35.964909    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx" podStartSLOduration=2.96489207 podStartE2EDuration="2.96489207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:35.916918649 +0000 UTC m=+968.883251993" watchObservedRunningTime="2026-10-31 03:23:35.96489207 +0000 UTC m=+968.931225413"
Oct 31 03:23:36.480444 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:36.480407    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:36.481754 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:36.481726    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 03:23:36.481880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:36.481772    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx" podUID="0dce71a3-b4c7-459d-92e8-c0515341abf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 03:23:36.985923 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:23:36.985880    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:23:37.480049 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:37.479959    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 03:23:37.480049 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:37.480013    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx" podUID="0dce71a3-b4c7-459d-92e8-c0515341abf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 03:23:38.412687 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:38.412645    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:38.480190 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:38.480155    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 03:23:38.480369 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:38.480225    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx" podUID="0dce71a3-b4c7-459d-92e8-c0515341abf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 03:23:38.621958 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:38.621908    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:38.917746 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:38.917709    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"]
Oct 31 03:23:38.921611 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:38.921591    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:38.972994 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:38.972954    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-dshm\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:38.973184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:38.973025    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-model-cache\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:38.973184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:38.973051    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc487b4-195a-42bb-9db6-4aefa3186c6b-tls-certs\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:38.973184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:38.973078    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-home\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:38.973184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:38.973153    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhx5p\" (UniqueName: \"kubernetes.io/projected/5fc487b4-195a-42bb-9db6-4aefa3186c6b-kube-api-access-jhx5p\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:38.973423 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:38.973232    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-tmp-dir\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:39.017410 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.017371    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7778d0\"/\"distinct-w0-7778d0-kserve-self-signed-certs\""
Oct 31 03:23:39.019757 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.019730    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"]
Oct 31 03:23:39.076174 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.076120    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-dshm\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:39.076501 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.076476    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-model-cache\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:39.076672 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.076656    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc487b4-195a-42bb-9db6-4aefa3186c6b-tls-certs\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:39.076857 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.076825    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-home\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:39.077159 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.077094    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhx5p\" (UniqueName: \"kubernetes.io/projected/5fc487b4-195a-42bb-9db6-4aefa3186c6b-kube-api-access-jhx5p\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:39.077991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.077968    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-tmp-dir\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:39.078150 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.078130    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-home\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:39.079193 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.077607    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-model-cache\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:39.079367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.078308    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-tmp-dir\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:39.079367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.079238    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:39.080592 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.080561    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-dshm\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:39.081363 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.081345    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc487b4-195a-42bb-9db6-4aefa3186c6b-tls-certs\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:39.131272 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.131234    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhx5p\" (UniqueName: \"kubernetes.io/projected/5fc487b4-195a-42bb-9db6-4aefa3186c6b-kube-api-access-jhx5p\") pod \"distinct-w0-7778d0-kserve-5559fddd48-vbg7d\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:39.234282 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.234008    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:39.480401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.480368    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 03:23:39.480841 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.480426    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx" podUID="0dce71a3-b4c7-459d-92e8-c0515341abf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 03:23:39.625913 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.625886    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"]
Oct 31 03:23:39.627453 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:23:39.627420    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fc487b4_195a_42bb_9db6_4aefa3186c6b.slice/crio-b489e377069f87d542573da9ae42c28fe74e1636ff086f6830d899f32037580c WatchSource:0}: Error finding container b489e377069f87d542573da9ae42c28fe74e1636ff086f6830d899f32037580c: Status 404 returned error can't find the container with id b489e377069f87d542573da9ae42c28fe74e1636ff086f6830d899f32037580c
Oct 31 03:23:39.780037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.779997    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d" event={"ID":"5fc487b4-195a-42bb-9db6-4aefa3186c6b","Type":"ContainerStarted","Data":"850fe939d0c5f9b08f02333a7f637faa672b007ed8b8464be971a6b4020532ae"}
Oct 31 03:23:39.780037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.780041    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d" event={"ID":"5fc487b4-195a-42bb-9db6-4aefa3186c6b","Type":"ContainerStarted","Data":"b489e377069f87d542573da9ae42c28fe74e1636ff086f6830d899f32037580c"}
Oct 31 03:23:39.886832 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:39.886728    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d" podStartSLOduration=1.886702608 podStartE2EDuration="1.886702608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:23:39.886599361 +0000 UTC m=+972.852932705" watchObservedRunningTime="2026-10-31 03:23:39.886702608 +0000 UTC m=+972.853035951"
Oct 31 03:23:40.255943 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:40.255896    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:40.480535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:40.480493    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 03:23:40.481042 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:40.480562    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx" podUID="0dce71a3-b4c7-459d-92e8-c0515341abf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 03:23:41.480160 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:41.480126    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 03:23:41.480359 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:41.480186    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx" podUID="0dce71a3-b4c7-459d-92e8-c0515341abf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 03:23:42.484713 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:42.484677    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:42.485236 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:42.485012    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:42.485716 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:42.485691    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:23:42.785207 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:42.784859    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:43.595759 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:43.595719    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:47.087945 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:47.087909    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:48.700917 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:48.700884    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:49.251342 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:49.251306    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:49.263731 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:49.263701    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:49.848173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:49.848140    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:23:53.580762 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:53.580711    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:23:58.483515 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:23:58.483477    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:03.383433 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:03.383395    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:06.985386 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:24:06.985350    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:24:08.280111 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:08.280077    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:13.154331 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:13.154219    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:13.364310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:13.364272    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:13.571318 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:13.571287    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:14.435636 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.435601    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"]
Oct 31 03:24:14.439850 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.439827    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.495023 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.494980    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"]
Oct 31 03:24:14.504265 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.504230    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-tmp-dir\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.504428 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.504319    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zr44\" (UniqueName: \"kubernetes.io/projected/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-kube-api-access-2zr44\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.504428 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.504364    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-home\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.504428 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.504407    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-model-cache\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.504576 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.504549    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-dshm\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.504622 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.504605    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-tls-certs\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.536326 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.536295    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"openshift-service-ca.crt\""
Oct 31 03:24:14.536510 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.536397    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"default-dockercfg-tmtvg\""
Oct 31 03:24:14.536561 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.536520    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"e2e-conflict-b-3b9043d1-kserve-self-signed-certs\""
Oct 31 03:24:14.537359 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.537343    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"kube-root-ca.crt\""
Oct 31 03:24:14.605519 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.605479    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-model-cache\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.605727 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.605539    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-dshm\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.605727 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.605641    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-tls-certs\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.605906 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.605817    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-tmp-dir\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.605906 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.605848    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zr44\" (UniqueName: \"kubernetes.io/projected/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-kube-api-access-2zr44\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.605906 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.605896    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-home\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.606067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.605993    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-model-cache\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.606172 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.606152    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-tmp-dir\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.606230 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.606212    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-home\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.607545 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.607517    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:14.609129 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.609101    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-tls-certs\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.609288 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.609274    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-dshm\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.660244 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.660209    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zr44\" (UniqueName: \"kubernetes.io/projected/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-kube-api-access-2zr44\") pod \"e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") " pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:14.752973 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:14.752937    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:15.135318 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:15.135284    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"]
Oct 31 03:24:15.135860 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:24:15.135835    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd7858d2_4464_4aa4_b7aa_fa750ac191ee.slice/crio-6df224936655efb5c02347cdb9e7cd0cec3db82ff78625155d06e84976fa6747 WatchSource:0}: Error finding container 6df224936655efb5c02347cdb9e7cd0cec3db82ff78625155d06e84976fa6747: Status 404 returned error can't find the container with id 6df224936655efb5c02347cdb9e7cd0cec3db82ff78625155d06e84976fa6747
Oct 31 03:24:15.978657 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:15.978618    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr" event={"ID":"dd7858d2-4464-4aa4-b7aa-fa750ac191ee","Type":"ContainerStarted","Data":"68223a25a29c072fdda7d5ca6f3f75432f688a89adfc3cf97c7e45edec1e6be9"}
Oct 31 03:24:15.979081 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:15.978669    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr" event={"ID":"dd7858d2-4464-4aa4-b7aa-fa750ac191ee","Type":"ContainerStarted","Data":"6df224936655efb5c02347cdb9e7cd0cec3db82ff78625155d06e84976fa6747"}
Oct 31 03:24:16.088268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:16.088207    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr" podStartSLOduration=2.088189046 podStartE2EDuration="2.088189046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:16.084646636 +0000 UTC m=+1009.050979981" watchObservedRunningTime="2026-10-31 03:24:16.088189046 +0000 UTC m=+1009.054522398"
Oct 31 03:24:16.753862 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:16.753824    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:18.190641 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:18.190600    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:23.178812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:23.178544    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:24.757260 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:24.757228    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:24.816257 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:24.816220    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:25.123701 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:25.123602    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:24:28.880131 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:28.880093    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:33.876369 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:33.876327    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:36.654265 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.654226    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"]
Oct 31 03:24:36.658215 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.658195    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.721367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.721334    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"]
Oct 31 03:24:36.749642 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.749610    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-model-cache\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.749814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.749663    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-tls-certs\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.749814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.749745    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-home\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.749897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.749874    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-tmp-dir\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.749932 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.749921    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxqm6\" (UniqueName: \"kubernetes.io/projected/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-kube-api-access-nxqm6\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.749969 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.749959    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-dshm\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.755320 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.755304    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7778d0\"/\"distinct-2-w0-7778d0-kserve-self-signed-certs\""
Oct 31 03:24:36.810025 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.809989    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:36.851192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.851154    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-model-cache\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.851388 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.851219    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-tls-certs\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.851388 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.851251    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-home\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.851388 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.851324    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-tmp-dir\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.851388 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.851375    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxqm6\" (UniqueName: \"kubernetes.io/projected/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-kube-api-access-nxqm6\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.851602 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.851431    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-dshm\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.851602 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.851589    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-model-cache\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.851712 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.851628    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-home\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.851712 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.851696    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-tmp-dir\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.853686 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.853658    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-dshm\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.853958 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.853941    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-tls-certs\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.904567 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.904496    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxqm6\" (UniqueName: \"kubernetes.io/projected/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-kube-api-access-nxqm6\") pod \"distinct-2-w0-7778d0-kserve-55b7c6c859-clml5\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") " pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.970481 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:36.970448    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:36.986307 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:24:36.986273    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:24:37.347977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:37.347948    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"]
Oct 31 03:24:37.349405 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:24:37.349374    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec7f8536_2b67_47fe_8f8e_7dd6ea39bb63.slice/crio-2ea336a312e5f980923d72f63f01bd7c0c943813754ac3ed74955dd3228e20bf WatchSource:0}: Error finding container 2ea336a312e5f980923d72f63f01bd7c0c943813754ac3ed74955dd3228e20bf: Status 404 returned error can't find the container with id 2ea336a312e5f980923d72f63f01bd7c0c943813754ac3ed74955dd3228e20bf
Oct 31 03:24:38.107618 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:38.107584    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" event={"ID":"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63","Type":"ContainerStarted","Data":"bbbe1d5e2c75d9d438d0cbc73d754b5ffd92b04905668821494dd4ac3de739b3"}
Oct 31 03:24:38.107618 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:38.107625    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" event={"ID":"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63","Type":"ContainerStarted","Data":"2ea336a312e5f980923d72f63f01bd7c0c943813754ac3ed74955dd3228e20bf"}
Oct 31 03:24:38.215971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:38.215911    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podStartSLOduration=2.215898102 podStartE2EDuration="2.215898102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:38.213728415 +0000 UTC m=+1031.180061760" watchObservedRunningTime="2026-10-31 03:24:38.215898102 +0000 UTC m=+1031.182231447"
Oct 31 03:24:38.931299 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:38.931270    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:38.971463 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:38.971431    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:40.493955 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.493922    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"]
Oct 31 03:24:40.497473 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.497456    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.594815 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.594748    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9360eb\"/\"distinct-2-w4-9360eb-kserve-self-signed-certs\""
Oct 31 03:24:40.595136 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.595112    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-model-cache\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.595244 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.595164    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-tmp-dir\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.595288 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.595269    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-dshm\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.595322 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.595307    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-home\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.595427 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.595403    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm579\" (UniqueName: \"kubernetes.io/projected/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-kube-api-access-bm579\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.595475 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.595461    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-tls-certs\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.597115 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.597096    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"]
Oct 31 03:24:40.664360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.664326    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:40.696560 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.696514    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-dshm\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.696777 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.696630    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-home\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.696777 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.696729    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm579\" (UniqueName: \"kubernetes.io/projected/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-kube-api-access-bm579\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.696927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.696783    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-tls-certs\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.696927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.696902    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-model-cache\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.697048 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.696954    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-tmp-dir\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.697150 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.697129    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-home\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.697261 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.697234    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-model-cache\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.697318 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.697265    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-tmp-dir\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.698967 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.698945    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-dshm\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.699165 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.699144    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-tls-certs\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.757243 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.757176    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm579\" (UniqueName: \"kubernetes.io/projected/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-kube-api-access-bm579\") pod \"distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") " pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:40.810675 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:40.810645    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:41.194884 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:41.194235    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"]
Oct 31 03:24:41.203612 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:24:41.203346    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c308b1a_70f2_45b0_9a25_d2f8501a3b1c.slice/crio-5d3e5474c472f27b3f72070e76e0d71dacafb31c6052520daf5a9bf7e4f646eb WatchSource:0}: Error finding container 5d3e5474c472f27b3f72070e76e0d71dacafb31c6052520daf5a9bf7e4f646eb: Status 404 returned error can't find the container with id 5d3e5474c472f27b3f72070e76e0d71dacafb31c6052520daf5a9bf7e4f646eb
Oct 31 03:24:42.182814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:42.172761    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" event={"ID":"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c","Type":"ContainerStarted","Data":"c18a3bb47e343cb6ef7024cf0be91a36c8b4f7dcc794624bf42c4ae546f5718a"}
Oct 31 03:24:42.182814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:42.172871    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" event={"ID":"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c","Type":"ContainerStarted","Data":"5d3e5474c472f27b3f72070e76e0d71dacafb31c6052520daf5a9bf7e4f646eb"}
Oct 31 03:24:42.291838 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:42.290759    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" podStartSLOduration=2.290738678 podStartE2EDuration="2.290738678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:42.290200776 +0000 UTC m=+1035.256534122" watchObservedRunningTime="2026-10-31 03:24:42.290738678 +0000 UTC m=+1035.257072022"
Oct 31 03:24:42.812939 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:42.812904    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:44.428347 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:44.428314    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:46.971646 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:46.971606    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:46.984263 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:46.984234    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:47.229047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:47.228960    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:24:47.384268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:47.384235    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:49.395778 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.395728    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:49.440054 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.440014    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"]
Oct 31 03:24:49.444181 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.444158    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.537283 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.537243    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.537429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.537304    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.537429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.537361    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.537516 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.537425    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d698f07e-e83e-4b52-9aba-1699cf8d1841-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.537516 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.537458    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tdfk\" (UniqueName: \"kubernetes.io/projected/d698f07e-e83e-4b52-9aba-1699cf8d1841-kube-api-access-7tdfk\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.537585 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.537521    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.537585 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.537565    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.537647 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.537600    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.537647 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.537638    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.639193 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.639146    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.639417 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.639217    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.639417 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.639271    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.639417 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.639298    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d698f07e-e83e-4b52-9aba-1699cf8d1841-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.639417 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.639333    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tdfk\" (UniqueName: \"kubernetes.io/projected/d698f07e-e83e-4b52-9aba-1699cf8d1841-kube-api-access-7tdfk\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.639417 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.639374    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.640144 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.639435    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.640144 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.639487    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.640144 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.639520    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.640144 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.640008    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.640384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.640308    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.640434 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.640414    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d698f07e-e83e-4b52-9aba-1699cf8d1841-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.640690 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.640647    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.640871 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.640819    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.642912 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.642863    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.643255 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.643202    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.644511 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.644464    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"]
Oct 31 03:24:49.691915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.691833    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.692059 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.691964    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tdfk\" (UniqueName: \"kubernetes.io/projected/d698f07e-e83e-4b52-9aba-1699cf8d1841-kube-api-access-7tdfk\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:49.758148 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:49.758111    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:50.172719 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:50.172688    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"]
Oct 31 03:24:50.176062 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:24:50.176033    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd698f07e_e83e_4b52_9aba_1699cf8d1841.slice/crio-2096caacc4da0dd617c8b0ea660d24cc50df034d750705db34b3067b0df465f8 WatchSource:0}: Error finding container 2096caacc4da0dd617c8b0ea660d24cc50df034d750705db34b3067b0df465f8: Status 404 returned error can't find the container with id 2096caacc4da0dd617c8b0ea660d24cc50df034d750705db34b3067b0df465f8
Oct 31 03:24:50.178528 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:50.178493    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:24:50.178610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:50.178560    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:24:50.178610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:50.178588    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:24:50.235199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:50.235168    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f" event={"ID":"d698f07e-e83e-4b52-9aba-1699cf8d1841","Type":"ContainerStarted","Data":"2096caacc4da0dd617c8b0ea660d24cc50df034d750705db34b3067b0df465f8"}
Oct 31 03:24:50.811716 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:50.811683    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:50.824276 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:50.824245    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:51.264399 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:51.264361    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f" event={"ID":"d698f07e-e83e-4b52-9aba-1699cf8d1841","Type":"ContainerStarted","Data":"d2e4a7aab6d453d49f946247b0551e87244d7fe796e2fe16545fa08e37980bf6"}
Oct 31 03:24:51.275953 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:51.275925    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:24:51.373078 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:51.373028    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f" podStartSLOduration=2.373009435 podStartE2EDuration="2.373009435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:51.37250997 +0000 UTC m=+1044.338843313" watchObservedRunningTime="2026-10-31 03:24:51.373009435 +0000 UTC m=+1044.339342779"
Oct 31 03:24:51.759184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:51.759129    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:51.760779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:51.760753    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 03:24:51.760934 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:51.760824    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f" podUID="d698f07e-e83e-4b52-9aba-1699cf8d1841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 03:24:52.759291 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:52.759248    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 03:24:52.759698 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:52.759311    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f" podUID="d698f07e-e83e-4b52-9aba-1699cf8d1841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 03:24:53.759301 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:53.759254    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 03:24:53.759839 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:53.759322    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f" podUID="d698f07e-e83e-4b52-9aba-1699cf8d1841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 03:24:54.605495 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:54.605454    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:24:54.761812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:54.759779    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 03:24:54.761812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:54.759877    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f" podUID="d698f07e-e83e-4b52-9aba-1699cf8d1841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 03:24:55.763811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:55.763172    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 03:24:55.763811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:55.763238    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f" podUID="d698f07e-e83e-4b52-9aba-1699cf8d1841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 03:24:56.758722 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:56.758669    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 03:24:56.758923 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:56.758760    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f" podUID="d698f07e-e83e-4b52-9aba-1699cf8d1841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 03:24:57.771507 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:57.771474    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:57.771977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:57.771773    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:57.772776 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:57.772758    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:24:59.818190 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:24:59.818159    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:04.700381 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:04.700338    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:06.984597 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:25:06.984562    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:25:09.601310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:09.601268    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:10.225047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:10.225014    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:14.616447 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:14.616414    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:19.642178 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:19.642146    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:22.877449 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:22.877415    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"]
Oct 31 03:25:22.881455 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:22.881437    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:22.955105 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:22.955062    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:22.955287 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:22.955127    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:22.955287 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:22.955211    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451148a6-ec1b-4620-b650-76f3a578d829-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:22.955287 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:22.955268    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451148a6-ec1b-4620-b650-76f3a578d829-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:22.955395 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:22.955373    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:22.955449 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:22.955436    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:22.955509 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:22.955496    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:22.955558 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:22.955515    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451148a6-ec1b-4620-b650-76f3a578d829-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:22.955558 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:22.955533    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvn82\" (UniqueName: \"kubernetes.io/projected/451148a6-ec1b-4620-b650-76f3a578d829-kube-api-access-qvn82\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.056305 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.056270    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.056495 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.056356    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.056495 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.056395    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.056495 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.056416    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451148a6-ec1b-4620-b650-76f3a578d829-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.056696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.056545    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvn82\" (UniqueName: \"kubernetes.io/projected/451148a6-ec1b-4620-b650-76f3a578d829-kube-api-access-qvn82\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.056696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.056613    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.056696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.056675    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.056898 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.056731    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451148a6-ec1b-4620-b650-76f3a578d829-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.056898 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.056801    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.056898 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.056822    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451148a6-ec1b-4620-b650-76f3a578d829-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.056898 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.056882    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.057095 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.056891    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.057234 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.057214    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451148a6-ec1b-4620-b650-76f3a578d829-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.057280 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.057251    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.059323 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.059298    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.059678 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.059657    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451148a6-ec1b-4620-b650-76f3a578d829-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.071996 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.071967    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"]
Oct 31 03:25:23.110543 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.110514    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451148a6-ec1b-4620-b650-76f3a578d829-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.110543 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.110526    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvn82\" (UniqueName: \"kubernetes.io/projected/451148a6-ec1b-4620-b650-76f3a578d829-kube-api-access-qvn82\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.194918 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.194826    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:23.620362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.620338    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"]
Oct 31 03:25:23.622426 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:25:23.622395    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451148a6_ec1b_4620_b650_76f3a578d829.slice/crio-301832fc2fa441c4f10c068456dcc6908df4c9051fbbecf41512f2d39c08bc88 WatchSource:0}: Error finding container 301832fc2fa441c4f10c068456dcc6908df4c9051fbbecf41512f2d39c08bc88: Status 404 returned error can't find the container with id 301832fc2fa441c4f10c068456dcc6908df4c9051fbbecf41512f2d39c08bc88
Oct 31 03:25:23.624915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.624881    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:23.624999 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.624949    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:23.624999 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:23.624978    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:24.464472 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:24.464437    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9" event={"ID":"451148a6-ec1b-4620-b650-76f3a578d829","Type":"ContainerStarted","Data":"47511370eba5edd3a27ebf7004b96c6d1b49607ad9675fcd03dc9e3cec3984d6"}
Oct 31 03:25:24.464947 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:24.464500    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9" event={"ID":"451148a6-ec1b-4620-b650-76f3a578d829","Type":"ContainerStarted","Data":"301832fc2fa441c4f10c068456dcc6908df4c9051fbbecf41512f2d39c08bc88"}
Oct 31 03:25:24.601155 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:24.601124    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:24.623645 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:24.623578    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9" podStartSLOduration=2.623558902 podStartE2EDuration="2.623558902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:24.62045228 +0000 UTC m=+1077.586785625" watchObservedRunningTime="2026-10-31 03:25:24.623558902 +0000 UTC m=+1077.589892248"
Oct 31 03:25:25.195000 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:25.194967    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:25.196202 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:25.196175    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 03:25:25.196436 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:25.196395    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9" podUID="451148a6-ec1b-4620-b650-76f3a578d829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 03:25:26.197938 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:26.197889    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 03:25:26.198521 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:26.198480    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9" podUID="451148a6-ec1b-4620-b650-76f3a578d829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 03:25:27.196038 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:27.196000    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 03:25:27.196239 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:27.196060    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9" podUID="451148a6-ec1b-4620-b650-76f3a578d829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 03:25:28.195181 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:28.195148    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 03:25:28.195669 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:28.195221    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9" podUID="451148a6-ec1b-4620-b650-76f3a578d829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 03:25:29.195170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:29.195131    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 03:25:29.195374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:29.195193    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9" podUID="451148a6-ec1b-4620-b650-76f3a578d829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 03:25:29.835405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:29.835365    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:30.195627 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:30.195516    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 31 03:25:30.195627 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:30.195576    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9" podUID="451148a6-ec1b-4620-b650-76f3a578d829" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 31 03:25:31.989678 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:31.989642    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:31.990243 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:31.990223    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:31.991029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:31.991006    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:25:34.705348 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:34.705310    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:35.824257 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:35.824226    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"]
Oct 31 03:25:36.079151 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:36.079067    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:36.984975 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:25:36.984937    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:25:37.998494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:37.998460    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"]
Oct 31 03:25:38.002568 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.002551    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.094859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.094813    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"]
Oct 31 03:25:38.174645 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.174608    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.174847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.174670    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.174847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.174747    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.174946 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.174858    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.174946 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.174895    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.174946 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.174925    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.174946 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.174944    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt48q\" (UniqueName: \"kubernetes.io/projected/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-kube-api-access-gt48q\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.175089 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.175043    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.175161 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.175139    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.276224 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.276142    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.276224 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.276201    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.276420 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.276234    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.276420 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.276260    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.276420 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.276308    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.276420 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.276331    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.276420 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.276358    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.276420 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.276380    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt48q\" (UniqueName: \"kubernetes.io/projected/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-kube-api-access-gt48q\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.276751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.276420    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.276751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.276575    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.276751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.276647    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.276931 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.276823    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.276931 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.276831    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.277015 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.276993    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.278670 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.278649    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.278992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.278974    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.330382 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.330359    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.330382 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.330378    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt48q\" (UniqueName: \"kubernetes.io/projected/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-kube-api-access-gt48q\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:38.615569 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:38.615488    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:39.032809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:39.032689    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"]
Oct 31 03:25:39.066069 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:25:39.066039    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fe5ead_d281_497c_ab7e_f3fd1d75e5bc.slice/crio-0e9c4f7b10e973f39455601fe9c4e5120fc05a41aa636c1e4e50a342b68728db WatchSource:0}: Error finding container 0e9c4f7b10e973f39455601fe9c4e5120fc05a41aa636c1e4e50a342b68728db: Status 404 returned error can't find the container with id 0e9c4f7b10e973f39455601fe9c4e5120fc05a41aa636c1e4e50a342b68728db
Oct 31 03:25:39.068963 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:39.068933    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:39.069039 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:39.068996    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:39.069039 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:39.069030    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:39.563005 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:39.562933    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2" event={"ID":"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc","Type":"ContainerStarted","Data":"8ec86705daab1b45faf1c0128c345f63dd7de1cd9ff6c9455d8bcfda77118cbf"}
Oct 31 03:25:39.563340 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:39.563303    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2" event={"ID":"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc","Type":"ContainerStarted","Data":"0e9c4f7b10e973f39455601fe9c4e5120fc05a41aa636c1e4e50a342b68728db"}
Oct 31 03:25:39.616616 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:39.616584    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:39.817045 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:39.816955    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:40.617993 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:40.617955    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 03:25:40.618415 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:40.618016    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2" podUID="37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 03:25:40.960196 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:40.960103    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:41.616261 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:41.616231    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 03:25:41.616445 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:41.616280    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2" podUID="37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 03:25:42.616424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:42.616386    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 03:25:42.616857 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:42.616443    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2" podUID="37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 03:25:43.616362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:43.616327    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 03:25:43.616542 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:43.616393    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2" podUID="37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 03:25:44.616770 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:44.616733    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 03:25:44.617306 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:44.616831    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2" podUID="37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 03:25:44.957868 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:44.945451    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:45.616717 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:45.616681    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 03:25:45.616924 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:45.616761    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2" podUID="37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 03:25:46.621012 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:46.620975    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:46.621503 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:46.621374    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:46.622095 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:46.622075    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:25:46.780304 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:46.780247    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2" podStartSLOduration=9.780232168 podStartE2EDuration="9.780232168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:39.768689318 +0000 UTC m=+1092.735022673" watchObservedRunningTime="2026-10-31 03:25:46.780232168 +0000 UTC m=+1099.746565511"
Oct 31 03:25:47.735722 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:47.735693    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:50.667388 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:50.667353    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:25:50.843542 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:50.843504    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr" podUID="dd7858d2-4464-4aa4-b7aa-fa750ac191ee" containerName="main" containerID="cri-o://68223a25a29c072fdda7d5ca6f3f75432f688a89adfc3cf97c7e45edec1e6be9" gracePeriod=45
Oct 31 03:25:51.116782 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.116756    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:25:51.206294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.206266    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-tmp-dir\") pod \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") "
Oct 31 03:25:51.206440 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.206347    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-home\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-home\") pod \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") "
Oct 31 03:25:51.206440 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.206405    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-model-cache\") pod \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") "
Oct 31 03:25:51.206440 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.206434    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-dshm\") pod \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") "
Oct 31 03:25:51.206574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.206466    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-kube-api-access-2zr44\" (UniqueName: \"kubernetes.io/projected/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-kube-api-access-2zr44\") pod \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") "
Oct 31 03:25:51.206574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.206510    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-tls-certs\") pod \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\" (UID: \"dd7858d2-4464-4aa4-b7aa-fa750ac191ee\") "
Oct 31 03:25:51.206574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.206546    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-tmp-dir" pod "dd7858d2-4464-4aa4-b7aa-fa750ac191ee" (UID: "dd7858d2-4464-4aa4-b7aa-fa750ac191ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:25:51.206733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.206559    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-home" pod "dd7858d2-4464-4aa4-b7aa-fa750ac191ee" (UID: "dd7858d2-4464-4aa4-b7aa-fa750ac191ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:25:51.206825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.206734    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-model-cache" pod "dd7858d2-4464-4aa4-b7aa-fa750ac191ee" (UID: "dd7858d2-4464-4aa4-b7aa-fa750ac191ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:25:51.207008 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.206989    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:51.207074 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.207037    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:51.207074 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.207052    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:51.208765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.208735    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-tls-certs" pod "dd7858d2-4464-4aa4-b7aa-fa750ac191ee" (UID: "dd7858d2-4464-4aa4-b7aa-fa750ac191ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:25:51.208765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.208742    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-dshm" pod "dd7858d2-4464-4aa4-b7aa-fa750ac191ee" (UID: "dd7858d2-4464-4aa4-b7aa-fa750ac191ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:25:51.208938 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.208773    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-kube-api-access-2zr44" pod "dd7858d2-4464-4aa4-b7aa-fa750ac191ee" (UID: "dd7858d2-4464-4aa4-b7aa-fa750ac191ee"). InnerVolumeSpecName "kube-api-access-2zr44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:25:51.308088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.308056    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:51.308088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.308083    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zr44\" (UniqueName: \"kubernetes.io/projected/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-kube-api-access-2zr44\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:51.308088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.308092    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd7858d2-4464-4aa4-b7aa-fa750ac191ee-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:25:51.680417 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.680378    4900 generic.go:356] "Generic (PLEG): container finished" podID="dd7858d2-4464-4aa4-b7aa-fa750ac191ee" containerID="68223a25a29c072fdda7d5ca6f3f75432f688a89adfc3cf97c7e45edec1e6be9" exitCode=0
Oct 31 03:25:51.680847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.680427    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"
Oct 31 03:25:51.680847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.680465    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr" event={"ID":"dd7858d2-4464-4aa4-b7aa-fa750ac191ee","Type":"ContainerDied","Data":"68223a25a29c072fdda7d5ca6f3f75432f688a89adfc3cf97c7e45edec1e6be9"}
Oct 31 03:25:51.680847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.680515    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr" event={"ID":"dd7858d2-4464-4aa4-b7aa-fa750ac191ee","Type":"ContainerDied","Data":"6df224936655efb5c02347cdb9e7cd0cec3db82ff78625155d06e84976fa6747"}
Oct 31 03:25:51.680847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.680532    4900 scope.go:122] "RemoveContainer" containerID="68223a25a29c072fdda7d5ca6f3f75432f688a89adfc3cf97c7e45edec1e6be9"
Oct 31 03:25:51.690222 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.690202    4900 scope.go:122] "RemoveContainer" containerID="68223a25a29c072fdda7d5ca6f3f75432f688a89adfc3cf97c7e45edec1e6be9"
Oct 31 03:25:51.690491 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:25:51.690472    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68223a25a29c072fdda7d5ca6f3f75432f688a89adfc3cf97c7e45edec1e6be9\": container with ID starting with 68223a25a29c072fdda7d5ca6f3f75432f688a89adfc3cf97c7e45edec1e6be9 not found: ID does not exist" containerID="68223a25a29c072fdda7d5ca6f3f75432f688a89adfc3cf97c7e45edec1e6be9"
Oct 31 03:25:51.690530 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.690500    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68223a25a29c072fdda7d5ca6f3f75432f688a89adfc3cf97c7e45edec1e6be9"} err="failed to get container status \"68223a25a29c072fdda7d5ca6f3f75432f688a89adfc3cf97c7e45edec1e6be9\": rpc error: code = NotFound desc = could not find container \"68223a25a29c072fdda7d5ca6f3f75432f688a89adfc3cf97c7e45edec1e6be9\": container with ID starting with 68223a25a29c072fdda7d5ca6f3f75432f688a89adfc3cf97c7e45edec1e6be9 not found: ID does not exist"
Oct 31 03:25:51.839223 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.839190    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"]
Oct 31 03:25:51.885116 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:51.885078    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr"]
Oct 31 03:25:52.033941 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:52.033897    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/e2e-conflict-b-3b9043d1-kserve-757bc99ff5-l54xr" podUID="dd7858d2-4464-4aa4-b7aa-fa750ac191ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:25:53.153413 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:53.153372    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd7858d2-4464-4aa4-b7aa-fa750ac191ee" path="/var/lib/kubelet/pods/dd7858d2-4464-4aa4-b7aa-fa750ac191ee/volumes"
Oct 31 03:25:55.706678 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:25:55.706634    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:00.811333 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:00.811292    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:03.898456 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:03.898426    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:05.826220 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:05.826184    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:06.984451 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:26:06.984415    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:26:10.921197 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:10.921157    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:16.130747 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:16.130709    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:18.428984 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:18.428947    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:18.649126 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:18.649078    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:21.093321 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:21.093281    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:26.187334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:26.187291    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:27.228977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.228928    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:26:27.305937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.305890    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:26:27.561505 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.561466    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"]
Oct 31 03:26:27.564803 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.564762    4900 cpu_manager.go:414] "containerMap: removing container" podUID="dd7858d2-4464-4aa4-b7aa-fa750ac191ee" containerName="main"
Oct 31 03:26:27.564975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.564958    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7858d2-4464-4aa4-b7aa-fa750ac191ee" containerName="main"
Oct 31 03:26:27.565219 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.565203    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7858d2-4464-4aa4-b7aa-fa750ac191ee" containerName="main"
Oct 31 03:26:27.575646 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.575622    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.676013 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.675983    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b59563\"/\"distinct-2-w2-b59563-kserve-self-signed-certs\""
Oct 31 03:26:27.713203 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.713170    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"]
Oct 31 03:26:27.734782 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.734741    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:27.749840 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.749804    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-model-cache\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.750110 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.750092    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0962c273-9b23-4c27-9e40-127a53cd190a-tls-certs\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.750219 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.750205    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-home\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.750337 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.750322    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-dshm\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.750427 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.750415    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-tmp-dir\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.750607 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.750592    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmd26\" (UniqueName: \"kubernetes.io/projected/0962c273-9b23-4c27-9e40-127a53cd190a-kube-api-access-zmd26\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.853199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.851755    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-model-cache\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.853499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.852253    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-model-cache\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.853499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.853478    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0962c273-9b23-4c27-9e40-127a53cd190a-tls-certs\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.853695 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.853535    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-home\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.853695 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.853614    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-dshm\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.853695 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.853642    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-tmp-dir\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.853921 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.853893    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-home\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.854172 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.854151    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-tmp-dir\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.854316 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.854298    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmd26\" (UniqueName: \"kubernetes.io/projected/0962c273-9b23-4c27-9e40-127a53cd190a-kube-api-access-zmd26\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.856469 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.856442    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-dshm\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.857185 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.857166    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0962c273-9b23-4c27-9e40-127a53cd190a-tls-certs\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:27.911779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:27.911748    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmd26\" (UniqueName: \"kubernetes.io/projected/0962c273-9b23-4c27-9e40-127a53cd190a-kube-api-access-zmd26\") pod \"distinct-2-w2-b59563-kserve-b97b5fb76-4v2th\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") " pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:28.032630 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:28.032592    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:28.429366 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:28.429335    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"]
Oct 31 03:26:28.434676 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:26:28.434650    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0962c273_9b23_4c27_9e40_127a53cd190a.slice/crio-e0e57c5630279140a5c8ba9b8a8e9b1c070ccdebaa0d603518e756858ace3c4c WatchSource:0}: Error finding container e0e57c5630279140a5c8ba9b8a8e9b1c070ccdebaa0d603518e756858ace3c4c: Status 404 returned error can't find the container with id e0e57c5630279140a5c8ba9b8a8e9b1c070ccdebaa0d603518e756858ace3c4c
Oct 31 03:26:28.913496 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:28.913446    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" event={"ID":"0962c273-9b23-4c27-9e40-127a53cd190a","Type":"ContainerStarted","Data":"48042c882f7cc6c0615c0cfc1181551412381d7442023a2e8e896d640f17130e"}
Oct 31 03:26:28.913496 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:28.913497    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" event={"ID":"0962c273-9b23-4c27-9e40-127a53cd190a","Type":"ContainerStarted","Data":"e0e57c5630279140a5c8ba9b8a8e9b1c070ccdebaa0d603518e756858ace3c4c"}
Oct 31 03:26:29.033002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:29.032972    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:31.283954 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.283904    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" podStartSLOduration=4.283889645 podStartE2EDuration="4.283889645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:29.022410754 +0000 UTC m=+1141.988744097" watchObservedRunningTime="2026-10-31 03:26:31.283889645 +0000 UTC m=+1144.250222988"
Oct 31 03:26:31.284349 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.284057    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"]
Oct 31 03:26:31.288214 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.288169    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.344402 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.344373    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"]
Oct 31 03:26:31.409109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.409074    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.409109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.409112    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.409305 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.409153    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfw4j\" (UniqueName: \"kubernetes.io/projected/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-kube-api-access-mfw4j\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.409353 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.409326    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.409528 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.409497    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.409596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.409584    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.409631 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.409609    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.409631 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.409627    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.409697 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.409647    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.511067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.511031    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.511271 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.511125    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.511271 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.511157    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.511271 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.511178    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.511450 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.511302    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.511450 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.511343    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.511450 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.511401    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.511450 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.511422    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.511649 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.511492    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfw4j\" (UniqueName: \"kubernetes.io/projected/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-kube-api-access-mfw4j\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.511649 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.511531    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.511649 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.511633    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.511759 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.511664    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.511876 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.511851    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.512389 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.512367    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.513513 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.513494    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.513585 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.513568    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.565952 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.565886    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.566080 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.565962    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfw4j\" (UniqueName: \"kubernetes.io/projected/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-kube-api-access-mfw4j\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:31.601866 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:31.601834    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:32.020744 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:32.020694    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"]
Oct 31 03:26:32.054318 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:26:32.054285    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf300bdd4_16f8_4e79_89b4_c1e1249c9d14.slice/crio-df53813021fe9b7a911e3231af64939b2dce25006f78df1afec78169b8de0dfe WatchSource:0}: Error finding container df53813021fe9b7a911e3231af64939b2dce25006f78df1afec78169b8de0dfe: Status 404 returned error can't find the container with id df53813021fe9b7a911e3231af64939b2dce25006f78df1afec78169b8de0dfe
Oct 31 03:26:32.056668 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:32.056632    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:32.056817 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:32.056727    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:32.056817 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:32.056776    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:32.233971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:32.233929    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:32.955811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:32.945860    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z" event={"ID":"f300bdd4-16f8-4e79-89b4-c1e1249c9d14","Type":"ContainerStarted","Data":"46f6527821eedb36499cf0ac565adfabe5ccc247425f0672282f3ec78502080d"}
Oct 31 03:26:32.955811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:32.945970    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z" event={"ID":"f300bdd4-16f8-4e79-89b4-c1e1249c9d14","Type":"ContainerStarted","Data":"df53813021fe9b7a911e3231af64939b2dce25006f78df1afec78169b8de0dfe"}
Oct 31 03:26:33.116042 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:33.115989    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z" podStartSLOduration=2.115974146 podStartE2EDuration="2.115974146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:33.113859654 +0000 UTC m=+1146.080192999" watchObservedRunningTime="2026-10-31 03:26:33.115974146 +0000 UTC m=+1146.082307490"
Oct 31 03:26:33.602045 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:33.602016    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:33.603371 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:33.603348    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 03:26:33.603484 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:33.603393    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z" podUID="f300bdd4-16f8-4e79-89b4-c1e1249c9d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 03:26:34.602904 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:34.602864    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 03:26:34.603305 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:34.602923    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z" podUID="f300bdd4-16f8-4e79-89b4-c1e1249c9d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 03:26:35.602288 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:35.602255    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 03:26:35.602465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:35.602306    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z" podUID="f300bdd4-16f8-4e79-89b4-c1e1249c9d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 03:26:36.602727 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:36.602686    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 03:26:36.603171 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:36.602752    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z" podUID="f300bdd4-16f8-4e79-89b4-c1e1249c9d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 03:26:36.986267 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:26:36.986231    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:26:37.363863 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:37.363323    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:37.602173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:37.602140    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 03:26:37.602375 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:37.602192    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z" podUID="f300bdd4-16f8-4e79-89b4-c1e1249c9d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 03:26:38.032719 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:38.032681    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:38.050380 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:38.050342    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:38.603151 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:38.603108    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 03:26:38.603320 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:38.603184    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z" podUID="f300bdd4-16f8-4e79-89b4-c1e1249c9d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 03:26:39.013050 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:39.013016    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:26:39.606530 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:39.606494    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:39.606922 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:39.606775    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:39.607404 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:39.607388    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:26:44.006222 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:44.006186    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:48.007509 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:48.007476    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:49.186452 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:49.186422    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:53.276865 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:53.276830    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:53.500580 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:53.500543    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:54.185437 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:54.185389    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:56.314379 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.314344    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"]
Oct 31 03:26:56.318405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.318384    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.368060 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.368030    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"default-dockercfg-tmtvg\""
Oct 31 03:26:56.413980 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.413939    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-model-cache\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.414154 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.414055    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-dshm\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.414154 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.414144    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-home\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.414250 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.414167    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-tmp-dir\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.414378 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.414358    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdjjz\" (UniqueName: \"kubernetes.io/projected/64487e0c-7344-45c3-af5e-23c2681cc64e-kube-api-access-vdjjz\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.414466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.414402    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64487e0c-7344-45c3-af5e-23c2681cc64e-tls-certs\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.414519 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.414475    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"kube-root-ca.crt\""
Oct 31 03:26:56.414519 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.414484    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"openshift-service-ca.crt\""
Oct 31 03:26:56.417081 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.417059    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"]
Oct 31 03:26:56.469324 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.469294    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d39257\"/\"distinct-w1-d39257-kserve-self-signed-certs\""
Oct 31 03:26:56.515167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.515132    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-model-cache\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.515365 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.515213    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-dshm\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.515365 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.515309    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-home\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.515365 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.515337    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-tmp-dir\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.515555 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.515536    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdjjz\" (UniqueName: \"kubernetes.io/projected/64487e0c-7344-45c3-af5e-23c2681cc64e-kube-api-access-vdjjz\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.515624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.515569    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-model-cache\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.515722 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.515700    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-home\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.515781 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.515722    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-tmp-dir\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.515998 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.515978    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64487e0c-7344-45c3-af5e-23c2681cc64e-tls-certs\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.517767 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.517747    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-dshm\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.518332 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.518316    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64487e0c-7344-45c3-af5e-23c2681cc64e-tls-certs\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.525068 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.525040    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:26:56.570254 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.570176    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdjjz\" (UniqueName: \"kubernetes.io/projected/64487e0c-7344-45c3-af5e-23c2681cc64e-kube-api-access-vdjjz\") pod \"distinct-w1-d39257-kserve-7b49987986-vbv8z\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") " pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:56.630688 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:56.630657    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:57.004923 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:57.004751    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"]
Oct 31 03:26:57.007584 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:26:57.007554    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64487e0c_7344_45c3_af5e_23c2681cc64e.slice/crio-be31c8b1cca59bb63496a5d37742ed176b90a0c9cccd1e9d079f8e4db1290850 WatchSource:0}: Error finding container be31c8b1cca59bb63496a5d37742ed176b90a0c9cccd1e9d079f8e4db1290850: Status 404 returned error can't find the container with id be31c8b1cca59bb63496a5d37742ed176b90a0c9cccd1e9d079f8e4db1290850
Oct 31 03:26:57.179855 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:57.179817    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" event={"ID":"64487e0c-7344-45c3-af5e-23c2681cc64e","Type":"ContainerStarted","Data":"d4e6e9c9e0a4351d46a04e64e9bdcbfb14a188ab1f9b5e3c2d535a740f2e518d"}
Oct 31 03:26:57.180035 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:57.179865    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" event={"ID":"64487e0c-7344-45c3-af5e-23c2681cc64e","Type":"ContainerStarted","Data":"be31c8b1cca59bb63496a5d37742ed176b90a0c9cccd1e9d079f8e4db1290850"}
Oct 31 03:26:57.297404 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:57.297309    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" podStartSLOduration=1.297295932 podStartE2EDuration="1.297295932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:57.295235191 +0000 UTC m=+1170.261568535" watchObservedRunningTime="2026-10-31 03:26:57.297295932 +0000 UTC m=+1170.263629275"
Oct 31 03:26:57.630894 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:57.630768    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:26:59.295323 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:26:59.295285    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:01.529736 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.529701    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"]
Oct 31 03:27:01.533967 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.533945    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.627185 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.627150    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"]
Oct 31 03:27:01.680374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.680336    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.680374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.680382    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.680645 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.680516    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.680645 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.680635    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.680762 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.680704    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.680825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.680811    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9b78\" (UniqueName: \"kubernetes.io/projected/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-kube-api-access-g9b78\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.680871 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.680858    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.680978 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.680961    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.681050 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.681001    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.792099 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.791991    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.793086 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.793036    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.793367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.793338    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.793496 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.793396    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.793496 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.793412    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.793618 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.793501    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.793618 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.793601    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.793728 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.793663    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.793728 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.793713    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9b78\" (UniqueName: \"kubernetes.io/projected/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-kube-api-access-g9b78\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.793900 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.793807    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.795372 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.794161    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.795372 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.795207    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.795372 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.794222    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.795372 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.794371    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.795372 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.794182    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.796266 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.796246    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.848184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.848152    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:01.848184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:01.848187    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9b78\" (UniqueName: \"kubernetes.io/projected/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-kube-api-access-g9b78\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:02.148395 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:02.148313    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:02.566572 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:02.566545    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"]
Oct 31 03:27:02.567969 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:27:02.567935    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3beebd5c_ad9f_4677_ac7c_e5a6ff4c6645.slice/crio-db14ae15aa41686e5ff6f2bcfbcc687baea929e7f35fb3b9a1a75b2fa4de4b88 WatchSource:0}: Error finding container db14ae15aa41686e5ff6f2bcfbcc687baea929e7f35fb3b9a1a75b2fa4de4b88: Status 404 returned error can't find the container with id db14ae15aa41686e5ff6f2bcfbcc687baea929e7f35fb3b9a1a75b2fa4de4b88
Oct 31 03:27:02.570382 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:02.570347    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:27:02.570503 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:02.570428    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:27:02.570503 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:02.570477    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:27:03.218230 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:03.218194    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs" event={"ID":"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645","Type":"ContainerStarted","Data":"dfd57efd3483a9f84b5d93ccb77da4a41dce0efcf090b30b14311484fe72211a"}
Oct 31 03:27:03.218406 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:03.218249    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs" event={"ID":"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645","Type":"ContainerStarted","Data":"db14ae15aa41686e5ff6f2bcfbcc687baea929e7f35fb3b9a1a75b2fa4de4b88"}
Oct 31 03:27:03.378258 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:03.378200    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs" podStartSLOduration=2.378184412 podStartE2EDuration="2.378184412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:03.376490248 +0000 UTC m=+1176.342823621" watchObservedRunningTime="2026-10-31 03:27:03.378184412 +0000 UTC m=+1176.344517755"
Oct 31 03:27:04.155813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:04.149997    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:04.174063 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:04.170398    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 03:27:04.174063 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:04.170459    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs" podUID="3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 03:27:04.767843 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:04.754471    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:05.231882 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:05.230900    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:05.247810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:05.247456    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 03:27:05.247810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:05.247514    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs" podUID="3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 03:27:05.307811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:05.306897    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 03:27:05.557870 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:05.544889    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:05.691816 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:05.690917    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:05.812890 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:05.811912    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 03:27:05.885944 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:05.885878    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp" podUID="e58cb630-6755-44c8-b068-2bbdb6d4fe15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 03:27:06.160036 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:06.159924    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 03:27:06.160036 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:06.159995    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs" podUID="3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 03:27:06.631609 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:06.631575    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:27:06.676085 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:06.676055    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:27:06.987510 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:27:06.987477    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:27:07.154813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.153782    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 03:27:07.154813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.153861    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs" podUID="3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 03:27:07.333214 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.333138    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:27:07.505502 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.505470    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"]
Oct 31 03:27:07.510352 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.510329    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.572540 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.572499    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"]
Oct 31 03:27:07.575698 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.575676    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.575794 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.575737    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.575794 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.575755    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.575936 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.575923    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.576012 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.575999    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.576050 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.576024    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.576094 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.576065    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.576144 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.576113    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzggw\" (UniqueName: \"kubernetes.io/projected/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-kube-api-access-bzggw\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.576233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.576216    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.677216 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.677128    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.677216 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.677194    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.677682 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.677283    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.677682 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.677304    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.677682 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.677354    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.677682 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.677412    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.677682 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.677442    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.677682 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.677465    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.677682 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.677487    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzggw\" (UniqueName: \"kubernetes.io/projected/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-kube-api-access-bzggw\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.678074 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.677882    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.678074 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.678001    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.678074 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.678048    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.678359 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.678335    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.678359 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.678354    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.679843 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.679819    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.680007 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.679988    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.731110 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.731076    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.731308 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.731285    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzggw\" (UniqueName: \"kubernetes.io/projected/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-kube-api-access-bzggw\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:07.824564 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:07.824533    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:08.160810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:08.160259    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 03:27:08.160810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:08.160351    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs" podUID="3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 03:27:08.271098 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:08.271021    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"]
Oct 31 03:27:08.273918 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:27:08.273892    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de8555d_e0c1_41de_ab04_5a26e8fc76bf.slice/crio-e6cfd7687b5e2c314f3f96b1b0d153b39b3f57e58709a93bc7f4f28805bd81ac WatchSource:0}: Error finding container e6cfd7687b5e2c314f3f96b1b0d153b39b3f57e58709a93bc7f4f28805bd81ac: Status 404 returned error can't find the container with id e6cfd7687b5e2c314f3f96b1b0d153b39b3f57e58709a93bc7f4f28805bd81ac
Oct 31 03:27:08.276640 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:08.276610    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:27:08.276741 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:08.276672    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:27:08.276741 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:08.276727    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:27:08.330513 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:08.330487    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv" event={"ID":"0de8555d-e0c1-41de-ab04-5a26e8fc76bf","Type":"ContainerStarted","Data":"e6cfd7687b5e2c314f3f96b1b0d153b39b3f57e58709a93bc7f4f28805bd81ac"}
Oct 31 03:27:09.149038 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:09.148825    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 03:27:09.149560 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:09.149530    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs" podUID="3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 03:27:09.348490 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:09.348453    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv" event={"ID":"0de8555d-e0c1-41de-ab04-5a26e8fc76bf","Type":"ContainerStarted","Data":"86c8e987c0c22b134f093c7f308e807d314178ba768357c16f380248630b230e"}
Oct 31 03:27:09.516833 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:09.516763    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv" podStartSLOduration=2.5167482249999997 podStartE2EDuration="2.516748225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:09.515662699 +0000 UTC m=+1182.481996044" watchObservedRunningTime="2026-10-31 03:27:09.516748225 +0000 UTC m=+1182.483081568"
Oct 31 03:27:09.825541 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:09.825462    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:09.827323 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:09.827300    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 03:27:09.827442 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:09.827345    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv" podUID="0de8555d-e0c1-41de-ab04-5a26e8fc76bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 03:27:10.408544 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:10.408512    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:10.825649 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:10.825608    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 03:27:10.825838 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:10.825686    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv" podUID="0de8555d-e0c1-41de-ab04-5a26e8fc76bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 03:27:11.167047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:11.166943    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:11.167224 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:11.167044    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs" podUID="3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:11.825636 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:11.825601    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 03:27:11.826057 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:11.825654    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv" podUID="0de8555d-e0c1-41de-ab04-5a26e8fc76bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 03:27:11.887943 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:11.887900    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:11.888887 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:11.888863    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:11.890196 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:11.890177    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:27:12.026887 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:12.026838    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:12.275838 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:12.275809    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:12.825851 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:12.825817    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 03:27:12.826320 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:12.825871    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv" podUID="0de8555d-e0c1-41de-ab04-5a26e8fc76bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 03:27:13.421623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:13.421582    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:13.659223 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:13.659126    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:13.825298 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:13.825261    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 03:27:13.825455 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:13.825322    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv" podUID="0de8555d-e0c1-41de-ab04-5a26e8fc76bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 03:27:14.825692 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:14.825651    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 03:27:14.826213 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:14.825700    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv" podUID="0de8555d-e0c1-41de-ab04-5a26e8fc76bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 03:27:15.716233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:15.716182    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:15.828975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:15.828944    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:15.829417 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:15.829398    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:15.830097 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:15.830078    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:27:20.604990 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:20.604954    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:22.526646 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.526614    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"]
Oct 31 03:27:22.529685 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.529667    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.628088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.628056    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"]
Oct 31 03:27:22.708711 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.708657    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1b3d8d-e144-44ed-8efa-193251bd599c-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.708906 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.708858    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.708906 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.708887    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.709001 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.708935    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.709035 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.709012    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.709035 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.709030    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj8nv\" (UniqueName: \"kubernetes.io/projected/2b1b3d8d-e144-44ed-8efa-193251bd599c-kube-api-access-wj8nv\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.709110 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.709055    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.709221 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.709200    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.709284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.709262    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.811093 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.811008    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.811093 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.811048    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.811093 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.811076    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.811313 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.811145    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.811357 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.811335    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8nv\" (UniqueName: \"kubernetes.io/projected/2b1b3d8d-e144-44ed-8efa-193251bd599c-kube-api-access-wj8nv\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.811408 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.811390    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.811466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.811449    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.811525 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.811458    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.811525 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.811493    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.811525 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.811519    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.811675 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.811593    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1b3d8d-e144-44ed-8efa-193251bd599c-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.811836 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.811778    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.811961 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.811913    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.812279 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.812260    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1b3d8d-e144-44ed-8efa-193251bd599c-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.814117 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.814087    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.814218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.814202    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.888436 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.888403    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:22.888704 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:22.888673    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj8nv\" (UniqueName: \"kubernetes.io/projected/2b1b3d8d-e144-44ed-8efa-193251bd599c-kube-api-access-wj8nv\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:23.142671 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:23.142588    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:23.562689 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:23.562658    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"]
Oct 31 03:27:23.563036 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:27:23.562858    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1b3d8d_e144_44ed_8efa_193251bd599c.slice/crio-8980576e52db555edef763bbe785a2d87c5e364c2d2320fe25e5372bbfc0150a WatchSource:0}: Error finding container 8980576e52db555edef763bbe785a2d87c5e364c2d2320fe25e5372bbfc0150a: Status 404 returned error can't find the container with id 8980576e52db555edef763bbe785a2d87c5e364c2d2320fe25e5372bbfc0150a
Oct 31 03:27:23.565419 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:23.565387    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:27:23.565521 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:23.565450    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:27:23.565521 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:23.565480    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:27:24.469107 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:24.469069    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" event={"ID":"2b1b3d8d-e144-44ed-8efa-193251bd599c","Type":"ContainerStarted","Data":"691e1d32cae276ddfeb59a09a47dd3542a8d48755fd6d8c60e1e10e225f8a55b"}
Oct 31 03:27:24.469325 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:24.469126    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" event={"ID":"2b1b3d8d-e144-44ed-8efa-193251bd599c","Type":"ContainerStarted","Data":"8980576e52db555edef763bbe785a2d87c5e364c2d2320fe25e5372bbfc0150a"}
Oct 31 03:27:24.633437 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:24.633384    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" podStartSLOduration=2.6333699250000002 podStartE2EDuration="2.633369925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:24.632046279 +0000 UTC m=+1197.598379623" watchObservedRunningTime="2026-10-31 03:27:24.633369925 +0000 UTC m=+1197.599703269"
Oct 31 03:27:25.164895 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:25.164859    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:27:25.165117 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:25.164919    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 03:27:25.169040 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:25.169018    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:25.876811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:25.873260    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:26.152877 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:26.151222    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:27:26.152877 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:26.151299    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 03:27:27.162813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:27.159295    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:27:27.185953 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:27.185899    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 03:27:27.621988 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:27.621933    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 03:27:27.664811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:27.655235    4900 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:27.799984 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:27.799925    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:27.857813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:27.846831    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d" podUID="5fc487b4-195a-42bb-9db6-4aefa3186c6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:27.928992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:27.928894    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp" podUID="e58cb630-6755-44c8-b068-2bbdb6d4fe15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:27.975813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:27.972315    4900 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:28.011471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.011416    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:28.151810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.151437    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:27:28.151810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.151508    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 03:27:28.229836 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.229719    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j" podUID="43738834-604f-4063-8c19-d210e1a44de7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 31 03:27:28.278104 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.278063    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:28.305813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.303176    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:28.374725 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.374680    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:28.385517 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.384766    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:27:28.425813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.423421    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 31 03:27:28.448873 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.448843    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:27:28.811816 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.810951    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:28.811816 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.811740    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:27:28.833830 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.830884    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:28.833830 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.830945    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f" podUID="d698f07e-e83e-4b52-9aba-1699cf8d1841" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:28.849766 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.847953    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d" podUID="5fc487b4-195a-42bb-9db6-4aefa3186c6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 31 03:27:28.849766 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.848417    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:27:28.946978 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.946919    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp" podUID="e58cb630-6755-44c8-b068-2bbdb6d4fe15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:28.948588 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:28.947714    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:27:29.072775 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.072664    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:27:29.076810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.073438    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:27:29.093813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.078561    4900 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 31 03:27:29.186811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.160573    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:27:29.186811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.160632    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 03:27:29.204049 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.203936    4900 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j" podUID="43738834-604f-4063-8c19-d210e1a44de7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.238242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.238203    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j" podUID="43738834-604f-4063-8c19-d210e1a44de7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.293908 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.293291    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.309028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.308983    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.327907 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.326172    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:27:29.327907 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.326234    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:27:29.327907 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.326270    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:27:29.385465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.385419    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.395143 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.395116    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:27:29.447506 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.447423    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.455823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.452037    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:27:29.691648 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.691545    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.695811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.692264    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:27:29.823389 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.823339    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:29.866859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.866818    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d" podUID="5fc487b4-195a-42bb-9db6-4aefa3186c6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.025813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:29.997089    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:30.044274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.044221    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp" podUID="e58cb630-6755-44c8-b068-2bbdb6d4fe15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.078875 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.075368    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.250070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.250036    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:27:30.643574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.643528    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 03:27:30.644329 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.304727    4900 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d" podUID="5fc487b4-195a-42bb-9db6-4aefa3186c6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.647897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.331972    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.655843 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.403920    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.656356 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.487732    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.660807 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.535326    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j" podUID="43738834-604f-4063-8c19-d210e1a44de7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 31 03:27:30.670259 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.304445    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.670761 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.535282    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-dffz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:30.670977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.670944    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2" podUID="aaf0d481-00ac-4262-8bc7-224f1ff8f54b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.694984 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.694934    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.845813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.841881    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 03:27:30.870821 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.869128    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d" podUID="5fc487b4-195a-42bb-9db6-4aefa3186c6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:30.916872 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:30.877722    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:31.072810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.071125    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp" podUID="e58cb630-6755-44c8-b068-2bbdb6d4fe15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:31.089810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.087026    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:31.162681 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.162643    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:27:31.346092 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.346011    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 03:27:31.615874 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.606689    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:31.653022 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.652116    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 03:27:31.682813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.675166    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:31.682813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.675113    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:31.682813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.675831    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j" podUID="43738834-604f-4063-8c19-d210e1a44de7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 31 03:27:31.682813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.675779    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 31 03:27:31.703934 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.703886    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 03:27:31.819950 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.819893    4900 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:31.885009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.858775    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:31.885753 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.885720    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d" podUID="5fc487b4-195a-42bb-9db6-4aefa3186c6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 31 03:27:31.915118 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.915063    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:31.961987 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.961935    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:31.975813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:31.971912    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv" podUID="0de8555d-e0c1-41de-ab04-5a26e8fc76bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:32.104811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:32.091937    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp" podUID="e58cb630-6755-44c8-b068-2bbdb6d4fe15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:32.129370 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:32.129332    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:27:32.165172 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:32.165071    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:27:32.165447 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:32.165398    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 03:27:32.681095 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:32.658864    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:32.681095 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:32.677976    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 31 03:27:32.681095 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:32.678319    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j" podUID="43738834-604f-4063-8c19-d210e1a44de7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:32.681095 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:32.678389    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 31 03:27:32.681095 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:32.679017    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 03:27:32.681095 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:32.681085    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 03:27:32.704809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:32.704523    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 03:27:32.795516 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:32.795423    4900 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:32.894929 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:32.894886    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:32.920935 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:32.920886    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d" podUID="5fc487b4-195a-42bb-9db6-4aefa3186c6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:33.157171 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.157124    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:27:33.189544 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.172733    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:27:33.191907 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.191859    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 03:27:33.317812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.310921    4900 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:33.339664 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.339628    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:27:33.672950 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.672901    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:33.685384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.682146    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j" podUID="43738834-604f-4063-8c19-d210e1a44de7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 31 03:27:33.685384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.682983    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded"
Oct 31 03:27:33.685384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.684258    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:33.717822 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.703045    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:27:33.718599 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.718577    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:27:33.731803 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.729187    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:33.757809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.732134    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 03:27:33.757809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.749050    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:27:33.825066 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.825037    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 03:27:33.839353 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.839326    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:27:33.839642 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.839627    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:27:33.843335 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:33.843297    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 03:27:34.143355 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:34.143327    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 03:27:34.143523 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:34.143372    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 03:27:35.206948 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:35.206908    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:35.374033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:35.374001    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:35.374439 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:35.374416    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:35.375190 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:35.375168    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:27:35.466773 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:35.466691    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:36.984972 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:27:36.984939    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:27:38.380215 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:38.380174    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:40.795708 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:40.795664    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:41.657617 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:41.657585    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:41.932679 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:41.932590    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:44.975225 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:44.975191    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:45.210488 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:45.210458    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:45.470106 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:45.470065    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"]
Oct 31 03:27:45.474184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:45.474010    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"
Oct 31 03:27:45.571016 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:45.570983    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6b3a29ec-dockercfg-fw6vz\""
Oct 31 03:27:45.571244 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:45.571031    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6b3a29ec\""
Oct 31 03:27:45.575890 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:45.575856    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"]
Oct 31 03:27:45.602586 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:45.602553    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjs7b\" (UniqueName: \"kubernetes.io/projected/d2e462aa-1784-4fa4-a7e2-025bcd7214fc-kube-api-access-pjs7b\") pod \"payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz\" (UID: \"d2e462aa-1784-4fa4-a7e2-025bcd7214fc\") " pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"
Oct 31 03:27:45.602769 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:45.602739    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e462aa-1784-4fa4-a7e2-025bcd7214fc-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz\" (UID: \"d2e462aa-1784-4fa4-a7e2-025bcd7214fc\") " pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"
Oct 31 03:27:45.704603 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:45.704570    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e462aa-1784-4fa4-a7e2-025bcd7214fc-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz\" (UID: \"d2e462aa-1784-4fa4-a7e2-025bcd7214fc\") " pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"
Oct 31 03:27:45.704817 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:45.704732    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjs7b\" (UniqueName: \"kubernetes.io/projected/d2e462aa-1784-4fa4-a7e2-025bcd7214fc-kube-api-access-pjs7b\") pod \"payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz\" (UID: \"d2e462aa-1784-4fa4-a7e2-025bcd7214fc\") " pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"
Oct 31 03:27:45.705306 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:45.705280    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e462aa-1784-4fa4-a7e2-025bcd7214fc-plugins-config-volume\") pod \"payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz\" (UID: \"d2e462aa-1784-4fa4-a7e2-025bcd7214fc\") " pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"
Oct 31 03:27:45.761305 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:45.761275    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjs7b\" (UniqueName: \"kubernetes.io/projected/d2e462aa-1784-4fa4-a7e2-025bcd7214fc-kube-api-access-pjs7b\") pod \"payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz\" (UID: \"d2e462aa-1784-4fa4-a7e2-025bcd7214fc\") " pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"
Oct 31 03:27:45.804095 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:45.804065    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"
Oct 31 03:27:46.179398 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:46.179368    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"]
Oct 31 03:27:46.179712 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:27:46.179468    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e462aa_1784_4fa4_a7e2_025bcd7214fc.slice/crio-292726ad3654799317246c61c2b85410f4c2d6f33a19d51a29f6d4ad915828da WatchSource:0}: Error finding container 292726ad3654799317246c61c2b85410f4c2d6f33a19d51a29f6d4ad915828da: Status 404 returned error can't find the container with id 292726ad3654799317246c61c2b85410f4c2d6f33a19d51a29f6d4ad915828da
Oct 31 03:27:46.288407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:46.288374    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz" event={"ID":"d2e462aa-1784-4fa4-a7e2-025bcd7214fc","Type":"ContainerStarted","Data":"ec3868015d96bda888618475034c42d85e114353a118af8ba47c965ff0907f33"}
Oct 31 03:27:46.288566 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:46.288430    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz" event={"ID":"d2e462aa-1784-4fa4-a7e2-025bcd7214fc","Type":"ContainerStarted","Data":"292726ad3654799317246c61c2b85410f4c2d6f33a19d51a29f6d4ad915828da"}
Oct 31 03:27:46.288566 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:46.288454    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"
Oct 31 03:27:46.399337 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:46.399287    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz" podStartSLOduration=1.399272755 podStartE2EDuration="1.399272755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:46.396325165 +0000 UTC m=+1219.362658510" watchObservedRunningTime="2026-10-31 03:27:46.399272755 +0000 UTC m=+1219.365606099"
Oct 31 03:27:46.715664 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:46.715626    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:48.479471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:48.479431    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:51.757583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:51.757552    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:27:54.652956 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:54.652904    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:54.698964 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:54.698859    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:54.863927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:54.863877    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 03:27:55.083900 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:55.083854    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:27:55.655922 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:55.655876    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:55.657709 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:55.657684    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:27:55.662882 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:55.662853    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j" podUID="43738834-604f-4063-8c19-d210e1a44de7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded"
Oct 31 03:27:55.700927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:55.700883    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:55.701603 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:55.701187    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:55.702539 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:55.701221    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded"
Oct 31 03:27:55.703315 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:55.701850    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:27:55.708771 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:55.708746    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:55.915869 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:55.881904    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 03:27:55.915869 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:55.882325    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:27:55.937377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:55.937336    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d" podUID="5fc487b4-195a-42bb-9db6-4aefa3186c6b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:55.988950 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:55.988915    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:27:55.989253 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:55.989207    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" podUID="9b030d08-9d78-4c66-8729-243e64177d52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.063820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:56.053870    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp" podUID="e58cb630-6755-44c8-b068-2bbdb6d4fe15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.094912 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:56.089061    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.094912 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:56.089436    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:27:56.146395 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:56.145131    4900 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.224407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:56.224322    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 03:27:56.668053 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:56.665871    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j" podUID="43738834-604f-4063-8c19-d210e1a44de7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.668053 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:56.666586    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:27:56.672909 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:56.672860    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.701906 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:56.701854    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded"
Oct 31 03:27:56.706883 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:56.706851    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:56.707457 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:56.707439    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:27:56.747606 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:56.745379    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 03:27:56.778367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:56.778328    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 03:27:56.915744 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:56.915701    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:27:57.047808 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:57.036089    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 03:27:57.416848 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:57.416744    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j"
Oct 31 03:27:57.417013 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:57.416848    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 03:27:57.417013 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:57.416896    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"
Oct 31 03:27:58.813189 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:27:58.813157    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:03.988674 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:03.988638    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"]
Oct 31 03:28:03.993690 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:03.993670    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.047411 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.047379    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 03:28:04.047587 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.047436    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 03:28:04.090761 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.090729    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 03:28:04.090964 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.090891    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-r7bff\""
Oct 31 03:28:04.095707 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.095681    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"]
Oct 31 03:28:04.110237 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.110207    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd6e56b-0a06-4a47-abea-12b07f41c2db-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.110376 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.110264    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.110433 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.110390    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.110527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.110509    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.110586 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.110551    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.110657 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.110638    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.110710 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.110683    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pztj\" (UniqueName: \"kubernetes.io/projected/4dd6e56b-0a06-4a47-abea-12b07f41c2db-kube-api-access-8pztj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.212165 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.212117    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.212357 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.212189    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pztj\" (UniqueName: \"kubernetes.io/projected/4dd6e56b-0a06-4a47-abea-12b07f41c2db-kube-api-access-8pztj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.212357 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.212316    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd6e56b-0a06-4a47-abea-12b07f41c2db-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.212486 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.212383    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.212486 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.212464    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.212648 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.212624    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.212724 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.212668    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.213258 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.213236    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.214663 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.214643    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd6e56b-0a06-4a47-abea-12b07f41c2db-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.214764 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.214699    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.215031 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.215012    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.215109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.215092    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.271949 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.271884    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pztj\" (UniqueName: \"kubernetes.io/projected/4dd6e56b-0a06-4a47-abea-12b07f41c2db-kube-api-access-8pztj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.272079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.272000    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrklv\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.306152 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.306124    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:04.623687 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.623609    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:04.727658 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.727630    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"]
Oct 31 03:28:04.728694 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:28:04.728667    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd6e56b_0a06_4a47_abea_12b07f41c2db.slice/crio-a17092633cc26d84f70eb74e7e0ac2553a958c793f137a0ed6c46c00a806dff1 WatchSource:0}: Error finding container a17092633cc26d84f70eb74e7e0ac2553a958c793f137a0ed6c46c00a806dff1: Status 404 returned error can't find the container with id a17092633cc26d84f70eb74e7e0ac2553a958c793f137a0ed6c46c00a806dff1
Oct 31 03:28:04.731219 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.731186    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:28:04.731318 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:04.731258    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:28:05.532889 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:05.532852    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv" event={"ID":"4dd6e56b-0a06-4a47-abea-12b07f41c2db","Type":"ContainerStarted","Data":"4a3cfe09600bf1e823e3409a6079815198d9a2ea1ecb8cb59600a5ccf4d32376"}
Oct 31 03:28:05.533303 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:05.532902    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv" event={"ID":"4dd6e56b-0a06-4a47-abea-12b07f41c2db","Type":"ContainerStarted","Data":"a17092633cc26d84f70eb74e7e0ac2553a958c793f137a0ed6c46c00a806dff1"}
Oct 31 03:28:05.533303 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:05.533130    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:05.535009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:05.534986    4900 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-qrklv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:28:05.535131 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:05.535030    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv" podUID="4dd6e56b-0a06-4a47-abea-12b07f41c2db" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:28:05.725153 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:05.725101    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv" podStartSLOduration=2.725088989 podStartE2EDuration="2.725088989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:05.689133618 +0000 UTC m=+1238.655466972" watchObservedRunningTime="2026-10-31 03:28:05.725088989 +0000 UTC m=+1238.691422333"
Oct 31 03:28:06.540237 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:06.540203    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:28:06.985223 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:28:06.985190    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:28:09.157800 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:09.157753    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"]
Oct 31 03:28:09.158291 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:09.158046    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs" podUID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerName="istio-proxy" containerID="cri-o://3a4d57284d835ded3e2007e7495c4c1ea25d4c6fe59282accf1db5e98a607c69" gracePeriod=30
Oct 31 03:28:09.579458 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:09.579421    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:13.294745 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:13.294709    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:28:13.295232 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:13.294773    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs" podUID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:28:14.482204 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.482175    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:28:14.502572 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.502540    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-credential-socket\") pod \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") "
Oct 31 03:28:14.502783 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.502589    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istiod-ca-cert\") pod \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") "
Oct 31 03:28:14.502783 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.502693    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-podinfo\") pod \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") "
Oct 31 03:28:14.502783 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.502756    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-workload-socket\") pod \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") "
Oct 31 03:28:14.502955 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.502907    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-credential-socket" pod "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" (UID: "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:28:14.503023 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.503004    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-token\" (UniqueName: \"kubernetes.io/projected/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-token\") pod \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") "
Oct 31 03:28:14.503073 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.503039    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-data\") pod \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") "
Oct 31 03:28:14.503120 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.503059    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-workload-socket" pod "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" (UID: "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:28:14.503120 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.503074    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-workload-certs\") pod \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") "
Oct 31 03:28:14.503210 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.503168    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-envoy\") pod \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") "
Oct 31 03:28:14.503210 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.503176    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istiod-ca-cert" pod "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" (UID: "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:14.503307 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.503247    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-kube-api-access-x846w\" (UniqueName: \"kubernetes.io/projected/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-kube-api-access-x846w\") pod \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\" (UID: \"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8\") "
Oct 31 03:28:14.503307 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.503272    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-workload-certs" pod "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" (UID: "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:28:14.503410 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.503352    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-data" pod "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" (UID: "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:28:14.504063 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.503956    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:14.504063 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.503984    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:14.504063 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.503999    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:14.504063 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.504013    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:14.504063 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.504027    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:14.506643 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.506614    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-token" pod "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" (UID: "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:14.506912 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.506875    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-kube-api-access-x846w" pod "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" (UID: "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8"). InnerVolumeSpecName "kube-api-access-x846w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:14.507596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.507552    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-podinfo" pod "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" (UID: "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:28:14.507596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.507563    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-envoy" pod "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" (UID: "ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:28:14.511242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.511208    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:14.590700 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.590663    4900 generic.go:356] "Generic (PLEG): container finished" podID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerID="3a4d57284d835ded3e2007e7495c4c1ea25d4c6fe59282accf1db5e98a607c69" exitCode=0
Oct 31 03:28:14.590921 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.590713    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"
Oct 31 03:28:14.590921 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.590735    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs" event={"ID":"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8","Type":"ContainerDied","Data":"3a4d57284d835ded3e2007e7495c4c1ea25d4c6fe59282accf1db5e98a607c69"}
Oct 31 03:28:14.590921 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.590777    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs" event={"ID":"ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8","Type":"ContainerDied","Data":"c142e37ac4f9bae48968bf0678de7ab4b6aa77f3c82ce4f9aa474387cbd2bce8"}
Oct 31 03:28:14.590921 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.590816    4900 scope.go:122] "RemoveContainer" containerID="3a4d57284d835ded3e2007e7495c4c1ea25d4c6fe59282accf1db5e98a607c69"
Oct 31 03:28:14.603962 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.603940    4900 scope.go:122] "RemoveContainer" containerID="98077e4376b1c2336b924aff4b23068ecad4a75e30a153ae0b0650b0453defa5"
Oct 31 03:28:14.611359 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.611328    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:14.611359 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.611360    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:14.611596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.611377    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:14.611596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.611394    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x846w\" (UniqueName: \"kubernetes.io/projected/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8-kube-api-access-x846w\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:14.624950 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.624922    4900 scope.go:122] "RemoveContainer" containerID="3a4d57284d835ded3e2007e7495c4c1ea25d4c6fe59282accf1db5e98a607c69"
Oct 31 03:28:14.625304 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:28:14.625282    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a4d57284d835ded3e2007e7495c4c1ea25d4c6fe59282accf1db5e98a607c69\": container with ID starting with 3a4d57284d835ded3e2007e7495c4c1ea25d4c6fe59282accf1db5e98a607c69 not found: ID does not exist" containerID="3a4d57284d835ded3e2007e7495c4c1ea25d4c6fe59282accf1db5e98a607c69"
Oct 31 03:28:14.625390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.625317    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a4d57284d835ded3e2007e7495c4c1ea25d4c6fe59282accf1db5e98a607c69"} err="failed to get container status \"3a4d57284d835ded3e2007e7495c4c1ea25d4c6fe59282accf1db5e98a607c69\": rpc error: code = NotFound desc = could not find container \"3a4d57284d835ded3e2007e7495c4c1ea25d4c6fe59282accf1db5e98a607c69\": container with ID starting with 3a4d57284d835ded3e2007e7495c4c1ea25d4c6fe59282accf1db5e98a607c69 not found: ID does not exist"
Oct 31 03:28:14.625390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.625347    4900 scope.go:122] "RemoveContainer" containerID="98077e4376b1c2336b924aff4b23068ecad4a75e30a153ae0b0650b0453defa5"
Oct 31 03:28:14.625645 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:28:14.625612    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98077e4376b1c2336b924aff4b23068ecad4a75e30a153ae0b0650b0453defa5\": container with ID starting with 98077e4376b1c2336b924aff4b23068ecad4a75e30a153ae0b0650b0453defa5 not found: ID does not exist" containerID="98077e4376b1c2336b924aff4b23068ecad4a75e30a153ae0b0650b0453defa5"
Oct 31 03:28:14.625758 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.625648    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98077e4376b1c2336b924aff4b23068ecad4a75e30a153ae0b0650b0453defa5"} err="failed to get container status \"98077e4376b1c2336b924aff4b23068ecad4a75e30a153ae0b0650b0453defa5\": rpc error: code = NotFound desc = could not find container \"98077e4376b1c2336b924aff4b23068ecad4a75e30a153ae0b0650b0453defa5\": container with ID starting with 98077e4376b1c2336b924aff4b23068ecad4a75e30a153ae0b0650b0453defa5 not found: ID does not exist"
Oct 31 03:28:14.743867 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.743820    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"]
Oct 31 03:28:14.791274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.791238    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-84wbs"]
Oct 31 03:28:14.793807 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:14.793763    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:15.008576 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:15.008479    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:15.206546 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:15.206510    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" path="/var/lib/kubelet/pods/ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8/volumes"
Oct 31 03:28:20.874623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:20.874590    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:25.922280 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:25.922240    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:28.565117 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:28.565070    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:28.873147 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:28.873052    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:31.315425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:31.315393    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:36.213887 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:36.213844    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:36.984894 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:28:36.984857    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:28:41.399694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:41.399494    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:46.946041 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:46.945999    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:48.813886 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:48.813847    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:51.895982 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:51.895929    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:57.207271 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:57.207238    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:28:57.818691 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:28:57.818656    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:02.603635 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:02.603596    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:04.062676 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:04.062626    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:04.455335 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:04.455200    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:04.691378 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:04.691346    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:06.987416 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:29:06.987380    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:29:07.818484 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:07.818439    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:08.621909 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:08.621870    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:09.367745 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:09.367711    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:09.591109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:09.591074    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:13.043376 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:13.043332    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:13.345638 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:13.345554    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:13.767650 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:13.767615    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:13.991090 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:13.991055    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:17.027644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:17.027609    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:17.254166 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:17.254125    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:18.823981 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:18.823935    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:19.746890 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:19.746841    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:23.708306 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:23.708272    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:29.507384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:29.507347    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:31.538278 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:31.538240    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:34.629380 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:34.629333    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:36.985890 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:29:36.985858    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:29:39.767253 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:39.767225    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:44.681256 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:44.681218    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:48.251407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:48.251359    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:48.891449 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:48.891412    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:49.158385 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:49.158300    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:49.390295 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:49.390262    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:49.629223 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:49.629190    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:49.857375 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:49.857116    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:50.084023 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:50.083973    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:53.438848 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:53.438772    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:53.683235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:53.683206    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:54.969777 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:54.969736    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"]
Oct 31 03:29:54.970331 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:54.970311    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerName="istio-proxy"
Oct 31 03:29:54.970377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:54.970331    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerName="istio-proxy"
Oct 31 03:29:54.970445 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:54.970436    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerName="istio-proxy"
Oct 31 03:29:54.970483 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:54.970451    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerName="istio-proxy"
Oct 31 03:29:54.973977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:54.973960    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.072427 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.072391    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-dd5b85-openshift-default-dockercfg-b24df\""
Oct 31 03:29:55.074921 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.074886    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"]
Oct 31 03:29:55.092239 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.092187    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.092239 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.092239    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4jkh\" (UniqueName: \"kubernetes.io/projected/88fcbedb-2530-4fc4-b9ec-b897333ead8e-kube-api-access-r4jkh\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.092466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.092445    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.092596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.092566    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.092861 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.092838    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.092960 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.092943    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.093004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.092994    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.093058 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.093024    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.093165 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.093145    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.195038 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.194999    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.195222 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.195051    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4jkh\" (UniqueName: \"kubernetes.io/projected/88fcbedb-2530-4fc4-b9ec-b897333ead8e-kube-api-access-r4jkh\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.195222 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.195116    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.195222 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.195185    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.195405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.195234    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.195405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.195297    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.195405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.195338    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.195405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.195363    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.195639 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.195426    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.195639 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.195624    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.195744 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.195728    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.196037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.196003    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.196037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.196031    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.196172 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.196104    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.197424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.197400    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.197642 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.197625    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.252369 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.252330    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.253302 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.253281    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4jkh\" (UniqueName: \"kubernetes.io/projected/88fcbedb-2530-4fc4-b9ec-b897333ead8e-kube-api-access-r4jkh\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.287506 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.287461    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:55.402288 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.402258    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:55.710354 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.710323    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"]
Oct 31 03:29:55.711974 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:29:55.711949    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88fcbedb_2530_4fc4_b9ec_b897333ead8e.slice/crio-6ce1a3343cbcc3ea6ea15ba209842f259938e82a8de517b9ce9c459f82f4f740 WatchSource:0}: Error finding container 6ce1a3343cbcc3ea6ea15ba209842f259938e82a8de517b9ce9c459f82f4f740: Status 404 returned error can't find the container with id 6ce1a3343cbcc3ea6ea15ba209842f259938e82a8de517b9ce9c459f82f4f740
Oct 31 03:29:55.714370 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.714336    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:55.714470 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.714414    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:55.714470 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:55.714455    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:56.307779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:56.307739    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm" event={"ID":"88fcbedb-2530-4fc4-b9ec-b897333ead8e","Type":"ContainerStarted","Data":"8b99b56c811f2134d1be98443c8a50f3c902b9d17933d66ed91b96e71a8ecb00"}
Oct 31 03:29:56.307779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:56.307813    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm" event={"ID":"88fcbedb-2530-4fc4-b9ec-b897333ead8e","Type":"ContainerStarted","Data":"6ce1a3343cbcc3ea6ea15ba209842f259938e82a8de517b9ce9c459f82f4f740"}
Oct 31 03:29:57.288077 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:57.288037    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:29:57.289710 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:57.289682    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 03:29:57.289874 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:57.289739    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm" podUID="88fcbedb-2530-4fc4-b9ec-b897333ead8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 03:29:58.288330 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:58.288293    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 03:29:58.288768 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:58.288346    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm" podUID="88fcbedb-2530-4fc4-b9ec-b897333ead8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 03:29:58.700867 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:58.699535    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:59.014236 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:59.014205    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:29:59.288542 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:59.288428    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 03:29:59.289202 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:29:59.289170    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm" podUID="88fcbedb-2530-4fc4-b9ec-b897333ead8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 03:30:00.194530 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.194474    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm" podStartSLOduration=6.194458536 podStartE2EDuration="6.194458536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:56.433479334 +0000 UTC m=+1349.399812678" watchObservedRunningTime="2026-10-31 03:30:00.194458536 +0000 UTC m=+1353.160791881"
Oct 31 03:30:00.194810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.194629    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"]
Oct 31 03:30:00.195464 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.195442    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerName="istio-proxy"
Oct 31 03:30:00.195464 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.195463    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab28c8e8-eb9f-41f2-b7f7-59c6c6a2d1a8" containerName="istio-proxy"
Oct 31 03:30:00.199148 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.199130    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"
Oct 31 03:30:00.282613 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.282576    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"]
Oct 31 03:30:00.288851 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.288822    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 03:30:00.289294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.288871    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm" podUID="88fcbedb-2530-4fc4-b9ec-b897333ead8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 03:30:00.298817 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.298760    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lfxw\" (UniqueName: \"kubernetes.io/projected/e9ee4069-ecb7-4955-b068-211a964968e1-kube-api-access-9lfxw\") pod \"maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw\" (UID: \"e9ee4069-ecb7-4955-b068-211a964968e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"
Oct 31 03:30:00.298958 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.298888    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ee4069-ecb7-4955-b068-211a964968e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw\" (UID: \"e9ee4069-ecb7-4955-b068-211a964968e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"
Oct 31 03:30:00.322732 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.322685    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:30:00.400099 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.400060    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lfxw\" (UniqueName: \"kubernetes.io/projected/e9ee4069-ecb7-4955-b068-211a964968e1-kube-api-access-9lfxw\") pod \"maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw\" (UID: \"e9ee4069-ecb7-4955-b068-211a964968e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"
Oct 31 03:30:00.400270 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.400117    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ee4069-ecb7-4955-b068-211a964968e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw\" (UID: \"e9ee4069-ecb7-4955-b068-211a964968e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"
Oct 31 03:30:00.400814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.400768    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ee4069-ecb7-4955-b068-211a964968e1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw\" (UID: \"e9ee4069-ecb7-4955-b068-211a964968e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"
Oct 31 03:30:00.461745 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.461664    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lfxw\" (UniqueName: \"kubernetes.io/projected/e9ee4069-ecb7-4955-b068-211a964968e1-kube-api-access-9lfxw\") pod \"maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw\" (UID: \"e9ee4069-ecb7-4955-b068-211a964968e1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"
Oct 31 03:30:00.513610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.513575    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"
Oct 31 03:30:00.906511 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.906483    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"]
Oct 31 03:30:00.910578 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:00.910557    4900 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:30:01.232376 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:01.232345    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:01.288891 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:01.288855    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 03:30:01.289294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:01.288917    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm" podUID="88fcbedb-2530-4fc4-b9ec-b897333ead8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 03:30:01.379949 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:01.379915    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw" event={"ID":"e9ee4069-ecb7-4955-b068-211a964968e1","Type":"ContainerStarted","Data":"bc1db288c9edb343e4c90017b2e1b72378ad4f6e4d11d223352fc73f09fd52f5"}
Oct 31 03:30:02.288307 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:02.288265    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 03:30:02.288490 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:02.288323    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm" podUID="88fcbedb-2530-4fc4-b9ec-b897333ead8e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 03:30:03.293058 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:03.293023    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:30:03.293544 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:03.293427    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:30:03.294084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:03.294062    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:30:06.305334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:06.305243    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:06.414342 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:06.414304    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw" event={"ID":"e9ee4069-ecb7-4955-b068-211a964968e1","Type":"ContainerStarted","Data":"74e1ec9358c7c30be0ce125849a6f5766071923caa6211a0b4b32f0f6c887260"}
Oct 31 03:30:06.520429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:06.520360    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw" podStartSLOduration=1.7182362530000002 podStartE2EDuration="6.520346188s" totalImagesPullingTime="4.802109935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:06.519994637 +0000 UTC m=+1359.486327980" watchObservedRunningTime="2026-10-31 03:30:06.520346188 +0000 UTC m=+1359.486679539"
Oct 31 03:30:06.987949 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:30:06.987913    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:30:07.422814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:07.422709    4900 generic.go:356] "Generic (PLEG): container finished" podID="e9ee4069-ecb7-4955-b068-211a964968e1" containerID="74e1ec9358c7c30be0ce125849a6f5766071923caa6211a0b4b32f0f6c887260" exitCode=60
Oct 31 03:30:07.423246 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:07.422826    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw" event={"ID":"e9ee4069-ecb7-4955-b068-211a964968e1","Type":"ContainerDied","Data":"74e1ec9358c7c30be0ce125849a6f5766071923caa6211a0b4b32f0f6c887260"}
Oct 31 03:30:07.423246 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:07.423099    4900 scope.go:122] "RemoveContainer" containerID="74e1ec9358c7c30be0ce125849a6f5766071923caa6211a0b4b32f0f6c887260"
Oct 31 03:30:08.430921 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:08.430884    4900 generic.go:356] "Generic (PLEG): container finished" podID="e9ee4069-ecb7-4955-b068-211a964968e1" containerID="06a2df392932aab20473906d4d396fba6ba19fa88844d33803bfef7d60ded97a" exitCode=60
Oct 31 03:30:08.430921 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:08.430973    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw" event={"ID":"e9ee4069-ecb7-4955-b068-211a964968e1","Type":"ContainerDied","Data":"06a2df392932aab20473906d4d396fba6ba19fa88844d33803bfef7d60ded97a"}
Oct 31 03:30:08.431518 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:08.431033    4900 scope.go:122] "RemoveContainer" containerID="74e1ec9358c7c30be0ce125849a6f5766071923caa6211a0b4b32f0f6c887260"
Oct 31 03:30:08.431518 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:08.431311    4900 scope.go:122] "RemoveContainer" containerID="06a2df392932aab20473906d4d396fba6ba19fa88844d33803bfef7d60ded97a"
Oct 31 03:30:08.431637 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:30:08.431617    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw_odh-ai-gateway-infra(e9ee4069-ecb7-4955-b068-211a964968e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw" podUID="e9ee4069-ecb7-4955-b068-211a964968e1"
Oct 31 03:30:09.447896 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:09.447663    4900 scope.go:122] "RemoveContainer" containerID="06a2df392932aab20473906d4d396fba6ba19fa88844d33803bfef7d60ded97a"
Oct 31 03:30:09.449299 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:30:09.449268    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw_odh-ai-gateway-infra(e9ee4069-ecb7-4955-b068-211a964968e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw" podUID="e9ee4069-ecb7-4955-b068-211a964968e1"
Oct 31 03:30:12.053823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:12.053757    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:14.420386 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:14.420353    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:17.434147 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:17.434111    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:21.146105 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:21.146068    4900 scope.go:122] "RemoveContainer" containerID="06a2df392932aab20473906d4d396fba6ba19fa88844d33803bfef7d60ded97a"
Oct 31 03:30:21.539672 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:21.539638    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw" event={"ID":"e9ee4069-ecb7-4955-b068-211a964968e1","Type":"ContainerStarted","Data":"29978091cbfa6111a47282178abe87d1de23faad87070e608aa7dd12be9e1dca"}
Oct 31 03:30:22.450854 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:22.450817    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:22.547564 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:22.547527    4900 generic.go:356] "Generic (PLEG): container finished" podID="e9ee4069-ecb7-4955-b068-211a964968e1" containerID="29978091cbfa6111a47282178abe87d1de23faad87070e608aa7dd12be9e1dca" exitCode=60
Oct 31 03:30:22.547735 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:22.547599    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw" event={"ID":"e9ee4069-ecb7-4955-b068-211a964968e1","Type":"ContainerDied","Data":"29978091cbfa6111a47282178abe87d1de23faad87070e608aa7dd12be9e1dca"}
Oct 31 03:30:22.547735 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:22.547658    4900 scope.go:122] "RemoveContainer" containerID="06a2df392932aab20473906d4d396fba6ba19fa88844d33803bfef7d60ded97a"
Oct 31 03:30:22.548096 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:22.548079    4900 scope.go:122] "RemoveContainer" containerID="29978091cbfa6111a47282178abe87d1de23faad87070e608aa7dd12be9e1dca"
Oct 31 03:30:22.548439 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:30:22.548416    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw_odh-ai-gateway-infra(e9ee4069-ecb7-4955-b068-211a964968e1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw" podUID="e9ee4069-ecb7-4955-b068-211a964968e1"
Oct 31 03:30:22.663717 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:22.663684    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"]
Oct 31 03:30:23.704660 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:23.704634    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"
Oct 31 03:30:23.775122 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:23.775091    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ee4069-ecb7-4955-b068-211a964968e1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ee4069-ecb7-4955-b068-211a964968e1-maas-api-service-ca\") pod \"e9ee4069-ecb7-4955-b068-211a964968e1\" (UID: \"e9ee4069-ecb7-4955-b068-211a964968e1\") "
Oct 31 03:30:23.775293 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:23.775277    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ee4069-ecb7-4955-b068-211a964968e1-kube-api-access-9lfxw\" (UniqueName: \"kubernetes.io/projected/e9ee4069-ecb7-4955-b068-211a964968e1-kube-api-access-9lfxw\") pod \"e9ee4069-ecb7-4955-b068-211a964968e1\" (UID: \"e9ee4069-ecb7-4955-b068-211a964968e1\") "
Oct 31 03:30:23.775533 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:23.775504    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ee4069-ecb7-4955-b068-211a964968e1-maas-api-service-ca" pod "e9ee4069-ecb7-4955-b068-211a964968e1" (UID: "e9ee4069-ecb7-4955-b068-211a964968e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:23.775854 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:23.775836    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ee4069-ecb7-4955-b068-211a964968e1-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:23.777285 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:23.777257    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ee4069-ecb7-4955-b068-211a964968e1-kube-api-access-9lfxw" pod "e9ee4069-ecb7-4955-b068-211a964968e1" (UID: "e9ee4069-ecb7-4955-b068-211a964968e1"). InnerVolumeSpecName "kube-api-access-9lfxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:23.877592 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:23.877502    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lfxw\" (UniqueName: \"kubernetes.io/projected/e9ee4069-ecb7-4955-b068-211a964968e1-kube-api-access-9lfxw\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:24.563725 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:24.563693    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"
Oct 31 03:30:24.563725 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:24.563718    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw" event={"ID":"e9ee4069-ecb7-4955-b068-211a964968e1","Type":"ContainerDied","Data":"bc1db288c9edb343e4c90017b2e1b72378ad4f6e4d11d223352fc73f09fd52f5"}
Oct 31 03:30:24.563983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:24.563783    4900 scope.go:122] "RemoveContainer" containerID="29978091cbfa6111a47282178abe87d1de23faad87070e608aa7dd12be9e1dca"
Oct 31 03:30:24.751447 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:24.751409    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"]
Oct 31 03:30:24.797682 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:24.797650    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-6b3a29ec-29890290-6zzfw"]
Oct 31 03:30:25.154860 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:25.154827    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ee4069-ecb7-4955-b068-211a964968e1" path="/var/lib/kubelet/pods/e9ee4069-ecb7-4955-b068-211a964968e1/volumes"
Oct 31 03:30:27.517953 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:27.517920    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:32.440930 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:32.440886    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:36.989207 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:30:36.989168    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:30:37.468551 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:37.468460    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:37.693861 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:37.693819    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:40.237279 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:40.237245    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:41.010099 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:41.010069    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:41.676130 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:41.676092    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:41.930496 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:41.930358    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:42.998565 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:42.998506    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:43.847539 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:43.847505    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:44.082207 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:44.082176    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:46.345805 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:46.345763    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:46.690041 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:46.689950    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:48.268037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:48.268006    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:49.482889 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:49.482852    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:49.749289 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:49.749207    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:49.991755 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:49.991722    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:53.643800 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:53.643764    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:56.757500 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:56.757446    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:56.974084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:56.974052    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:30:58.647264 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:30:58.647233    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:03.602145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:03.602113    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:06.985889 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:31:06.985847    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:31:08.724436 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:08.724398    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:10.632152 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.632120    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"]
Oct 31 03:31:10.632722 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.632706    4900 cpu_manager.go:414] "containerMap: removing container" podUID="e9ee4069-ecb7-4955-b068-211a964968e1" containerName="cleanup"
Oct 31 03:31:10.632779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.632724    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ee4069-ecb7-4955-b068-211a964968e1" containerName="cleanup"
Oct 31 03:31:10.632779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.632738    4900 cpu_manager.go:414] "containerMap: removing container" podUID="e9ee4069-ecb7-4955-b068-211a964968e1" containerName="cleanup"
Oct 31 03:31:10.632779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.632743    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ee4069-ecb7-4955-b068-211a964968e1" containerName="cleanup"
Oct 31 03:31:10.632906 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.632862    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ee4069-ecb7-4955-b068-211a964968e1" containerName="cleanup"
Oct 31 03:31:10.632906 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.632870    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ee4069-ecb7-4955-b068-211a964968e1" containerName="cleanup"
Oct 31 03:31:10.632906 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.632879    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ee4069-ecb7-4955-b068-211a964968e1" containerName="cleanup"
Oct 31 03:31:10.636300 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.636282    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.715458 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.715427    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"]
Oct 31 03:31:10.735180 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.735141    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.735345 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.735195    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.735388 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.735330    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.735422 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.735385    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.735457 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.735440    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.735489 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.735479    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.735525 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.735515    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.735675 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.735649    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pj22\" (UniqueName: \"kubernetes.io/projected/dcc714a7-95eb-4aab-9974-09e37b18e6c9-kube-api-access-9pj22\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.735721 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.735703    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.837091 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.837051    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.837324 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.837101    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.837324 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.837165    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.837324 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.837201    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.837324 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.837232    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.837324 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.837287    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.837599 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.837335    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.837599 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.837381    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pj22\" (UniqueName: \"kubernetes.io/projected/dcc714a7-95eb-4aab-9974-09e37b18e6c9-kube-api-access-9pj22\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.837599 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.837422    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.837599 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.837549    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.837833 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.837655    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.837833 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.837705    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.837923 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.837827    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.838017 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.837985    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.839531 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.839507    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.839862 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.839846    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.890721 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.890646    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.890721 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.890717    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pj22\" (UniqueName: \"kubernetes.io/projected/dcc714a7-95eb-4aab-9974-09e37b18e6c9-kube-api-access-9pj22\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:10.949615 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:10.949585    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:11.365307 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:11.365283    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"]
Oct 31 03:31:11.366459 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:31:11.366436    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc714a7_95eb_4aab_9974_09e37b18e6c9.slice/crio-e6338f3a19083748a5a9c9451765ef44141d2b79af29d21d927e4aac562660fd WatchSource:0}: Error finding container e6338f3a19083748a5a9c9451765ef44141d2b79af29d21d927e4aac562660fd: Status 404 returned error can't find the container with id e6338f3a19083748a5a9c9451765ef44141d2b79af29d21d927e4aac562660fd
Oct 31 03:31:11.368772 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:11.368734    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:11.368879 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:11.368820    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:11.368879 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:11.368853    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:11.873329 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:11.873287    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n" event={"ID":"dcc714a7-95eb-4aab-9974-09e37b18e6c9","Type":"ContainerStarted","Data":"c954017a5e176be6944ea4718890da619129645c57257b42eca514fa5458bdcc"}
Oct 31 03:31:11.873988 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:11.873954    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n" event={"ID":"dcc714a7-95eb-4aab-9974-09e37b18e6c9","Type":"ContainerStarted","Data":"e6338f3a19083748a5a9c9451765ef44141d2b79af29d21d927e4aac562660fd"}
Oct 31 03:31:11.952939 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:11.952895    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:11.985549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:11.985495    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n" podStartSLOduration=1.985481606 podStartE2EDuration="1.985481606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:11.984003419 +0000 UTC m=+1424.950336764" watchObservedRunningTime="2026-10-31 03:31:11.985481606 +0000 UTC m=+1424.951814950"
Oct 31 03:31:12.956053 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:12.956014    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 03:31:12.956674 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:12.956642    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n" podUID="dcc714a7-95eb-4aab-9974-09e37b18e6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 03:31:13.951289 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:13.951252    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 03:31:13.951575 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:13.951547    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n" podUID="dcc714a7-95eb-4aab-9974-09e37b18e6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 03:31:14.074815 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:14.069261    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:14.955812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:14.953818    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 03:31:14.955812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:14.953884    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n" podUID="dcc714a7-95eb-4aab-9974-09e37b18e6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 03:31:15.953269 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:15.953226    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 03:31:15.953935 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:15.953900    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n" podUID="dcc714a7-95eb-4aab-9974-09e37b18e6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 03:31:16.346814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:16.344774    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:16.952233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:16.952193    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 03:31:16.952542 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:16.952508    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n" podUID="dcc714a7-95eb-4aab-9974-09e37b18e6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 03:31:17.950741 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:17.950704    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 03:31:17.951168 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:17.950754    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n" podUID="dcc714a7-95eb-4aab-9974-09e37b18e6c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 03:31:18.244712 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:18.244671    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"]
Oct 31 03:31:18.700415 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:18.700323    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:18.927261 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:18.927221    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:18.955219 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:18.955134    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:18.955820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:18.955772    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:18.956531 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:18.956509    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:31:19.150614 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:19.150581    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:22.370059 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:22.370022    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:23.246049 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:23.245993    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz" podUID="d2e462aa-1784-4fa4-a7e2-025bcd7214fc" containerName="payload-processing" containerID="cri-o://ec3868015d96bda888618475034c42d85e114353a118af8ba47c965ff0907f33" gracePeriod=25
Oct 31 03:31:24.228137 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:24.228096    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:27.295920 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:27.295879    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz" podUID="d2e462aa-1784-4fa4-a7e2-025bcd7214fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:31:29.525612 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:29.525573    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:30.259836 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:30.259771    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:30.978235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:30.978204    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:31.570226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:31.570178    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:31.791473 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:31.791437    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:32.273560 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:32.273521    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:32.892044 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:32.892001    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"]
Oct 31 03:31:32.893212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:32.893185    4900 cpu_manager.go:414] "containerMap: removing container" podUID="e9ee4069-ecb7-4955-b068-211a964968e1" containerName="cleanup"
Oct 31 03:31:32.893381 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:32.893357    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ee4069-ecb7-4955-b068-211a964968e1" containerName="cleanup"
Oct 31 03:31:32.898447 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:32.898422    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:32.949889 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:32.949858    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"]
Oct 31 03:31:32.996116 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:32.996079    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5effde-openshift-default-dockercfg-7cjnz\""
Oct 31 03:31:33.052017 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.051972    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-token\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.052197 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.052057    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-workload-socket\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.052197 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.052114    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzts9\" (UniqueName: \"kubernetes.io/projected/676e6a62-14d6-42a5-81d7-5b08d33de624-kube-api-access-tzts9\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.052308 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.052227    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-credential-socket\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.052308 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.052281    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-workload-certs\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.052401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.052387    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-data\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.052452 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.052432    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676e6a62-14d6-42a5-81d7-5b08d33de624-istiod-ca-cert\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.052587 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.052564    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-envoy\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.052690 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.052631    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-podinfo\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.058657 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.058608    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:33.155881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.155719    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-data\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.155881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.155828    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676e6a62-14d6-42a5-81d7-5b08d33de624-istiod-ca-cert\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.156083 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.155916    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-envoy\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.156083 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.155988    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-podinfo\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.156165 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.156137    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-token\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.156202 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.156185    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-workload-socket\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.156317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.156261    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzts9\" (UniqueName: \"kubernetes.io/projected/676e6a62-14d6-42a5-81d7-5b08d33de624-kube-api-access-tzts9\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.156317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.156302    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-credential-socket\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.156445 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.156369    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-workload-certs\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.159007 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.158967    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-data\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.159971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.159936    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-credential-socket\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.160133 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.159773    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-workload-socket\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.160133 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.160070    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676e6a62-14d6-42a5-81d7-5b08d33de624-istiod-ca-cert\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.160476 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.160431    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-workload-certs\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.162286 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.162261    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-podinfo\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.173533 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.173461    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-envoy\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.211577 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.211545    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-token\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.211739 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.211611    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzts9\" (UniqueName: \"kubernetes.io/projected/676e6a62-14d6-42a5-81d7-5b08d33de624-kube-api-access-tzts9\") pod \"e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") " pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.218656 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.218622    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:33.302840 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.298563    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:33.700247 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.700217    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"]
Oct 31 03:31:33.701044 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:31:33.701013    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676e6a62_14d6_42a5_81d7_5b08d33de624.slice/crio-152e5b06bae66e53be3c7cd1348996cfbcc68fe7ec5fe3bc93549bc386cea7c9 WatchSource:0}: Error finding container 152e5b06bae66e53be3c7cd1348996cfbcc68fe7ec5fe3bc93549bc386cea7c9: Status 404 returned error can't find the container with id 152e5b06bae66e53be3c7cd1348996cfbcc68fe7ec5fe3bc93549bc386cea7c9
Oct 31 03:31:33.703551 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.703517    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:33.703625 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.703584    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:33.703625 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:33.703609    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:34.037044 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:34.036992    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4" event={"ID":"676e6a62-14d6-42a5-81d7-5b08d33de624","Type":"ContainerStarted","Data":"3b7c2b4735ee0268eef08896dff22ba6591ee85777f245f15af48c168915be90"}
Oct 31 03:31:34.037257 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:34.037071    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4" event={"ID":"676e6a62-14d6-42a5-81d7-5b08d33de624","Type":"ContainerStarted","Data":"152e5b06bae66e53be3c7cd1348996cfbcc68fe7ec5fe3bc93549bc386cea7c9"}
Oct 31 03:31:34.148141 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:34.148029    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4" podStartSLOduration=2.148015947 podStartE2EDuration="2.148015947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:34.146466327 +0000 UTC m=+1447.112799676" watchObservedRunningTime="2026-10-31 03:31:34.148015947 +0000 UTC m=+1447.114349314"
Oct 31 03:31:34.219223 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:34.219189    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:34.225045 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:34.225019    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:35.043589 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:35.043549    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:35.044737 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:35.044703    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:31:35.151893 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:35.151852    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:35.371448 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:35.371313    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:35.582528 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:35.582487    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:36.787325 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:36.787289    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:36.986024 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:31:36.985975    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:31:37.020695 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:37.020661    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:37.296010 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:37.295976    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz" podUID="d2e462aa-1784-4fa4-a7e2-025bcd7214fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:31:37.466321 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:37.466288    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:37.804698 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:37.804663    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:40.999864 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:40.994498    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:42.001022 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:42.000986    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:42.223280 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:42.223246    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:43.526472 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:43.526444    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"
Oct 31 03:31:43.643444 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:43.643410    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2e462aa-1784-4fa4-a7e2-025bcd7214fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e462aa-1784-4fa4-a7e2-025bcd7214fc-plugins-config-volume\") pod \"d2e462aa-1784-4fa4-a7e2-025bcd7214fc\" (UID: \"d2e462aa-1784-4fa4-a7e2-025bcd7214fc\") "
Oct 31 03:31:43.643618 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:43.643530    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2e462aa-1784-4fa4-a7e2-025bcd7214fc-kube-api-access-pjs7b\" (UniqueName: \"kubernetes.io/projected/d2e462aa-1784-4fa4-a7e2-025bcd7214fc-kube-api-access-pjs7b\") pod \"d2e462aa-1784-4fa4-a7e2-025bcd7214fc\" (UID: \"d2e462aa-1784-4fa4-a7e2-025bcd7214fc\") "
Oct 31 03:31:43.643795 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:43.643763    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2e462aa-1784-4fa4-a7e2-025bcd7214fc-plugins-config-volume" pod "d2e462aa-1784-4fa4-a7e2-025bcd7214fc" (UID: "d2e462aa-1784-4fa4-a7e2-025bcd7214fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:31:43.644256 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:43.644234    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2e462aa-1784-4fa4-a7e2-025bcd7214fc-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:43.645656 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:43.645625    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e462aa-1784-4fa4-a7e2-025bcd7214fc-kube-api-access-pjs7b" pod "d2e462aa-1784-4fa4-a7e2-025bcd7214fc" (UID: "d2e462aa-1784-4fa4-a7e2-025bcd7214fc"). InnerVolumeSpecName "kube-api-access-pjs7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:43.746001 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:43.745968    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjs7b\" (UniqueName: \"kubernetes.io/projected/d2e462aa-1784-4fa4-a7e2-025bcd7214fc-kube-api-access-pjs7b\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:44.114125 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:44.114089    4900 generic.go:356] "Generic (PLEG): container finished" podID="d2e462aa-1784-4fa4-a7e2-025bcd7214fc" containerID="ec3868015d96bda888618475034c42d85e114353a118af8ba47c965ff0907f33" exitCode=0
Oct 31 03:31:44.114349 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:44.114134    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz" event={"ID":"d2e462aa-1784-4fa4-a7e2-025bcd7214fc","Type":"ContainerDied","Data":"ec3868015d96bda888618475034c42d85e114353a118af8ba47c965ff0907f33"}
Oct 31 03:31:44.114349 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:44.114144    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"
Oct 31 03:31:44.114349 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:44.114215    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz" event={"ID":"d2e462aa-1784-4fa4-a7e2-025bcd7214fc","Type":"ContainerDied","Data":"292726ad3654799317246c61c2b85410f4c2d6f33a19d51a29f6d4ad915828da"}
Oct 31 03:31:44.114349 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:44.114245    4900 scope.go:122] "RemoveContainer" containerID="ec3868015d96bda888618475034c42d85e114353a118af8ba47c965ff0907f33"
Oct 31 03:31:44.124997 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:44.124967    4900 scope.go:122] "RemoveContainer" containerID="ec3868015d96bda888618475034c42d85e114353a118af8ba47c965ff0907f33"
Oct 31 03:31:44.125304 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:31:44.125278    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec3868015d96bda888618475034c42d85e114353a118af8ba47c965ff0907f33\": container with ID starting with ec3868015d96bda888618475034c42d85e114353a118af8ba47c965ff0907f33 not found: ID does not exist" containerID="ec3868015d96bda888618475034c42d85e114353a118af8ba47c965ff0907f33"
Oct 31 03:31:44.125400 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:44.125309    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec3868015d96bda888618475034c42d85e114353a118af8ba47c965ff0907f33"} err="failed to get container status \"ec3868015d96bda888618475034c42d85e114353a118af8ba47c965ff0907f33\": rpc error: code = NotFound desc = could not find container \"ec3868015d96bda888618475034c42d85e114353a118af8ba47c965ff0907f33\": container with ID starting with ec3868015d96bda888618475034c42d85e114353a118af8ba47c965ff0907f33 not found: ID does not exist"
Oct 31 03:31:44.269344 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:44.269312    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"]
Oct 31 03:31:44.315448 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:44.315413    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6b3a29ec-76cff78f69-cp2fz"]
Oct 31 03:31:44.702358 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:44.702316    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:44.958630 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:44.958481    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:45.136403 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.136367    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"]
Oct 31 03:31:45.137690 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.137661    4900 cpu_manager.go:414] "containerMap: removing container" podUID="d2e462aa-1784-4fa4-a7e2-025bcd7214fc" containerName="payload-processing"
Oct 31 03:31:45.137902 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.137877    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2e462aa-1784-4fa4-a7e2-025bcd7214fc" containerName="payload-processing"
Oct 31 03:31:45.138153 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.138078    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2e462aa-1784-4fa4-a7e2-025bcd7214fc" containerName="payload-processing"
Oct 31 03:31:45.158571 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.157503    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"
Oct 31 03:31:45.179743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.179421    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2e462aa-1784-4fa4-a7e2-025bcd7214fc" path="/var/lib/kubelet/pods/d2e462aa-1784-4fa4-a7e2-025bcd7214fc/volumes"
Oct 31 03:31:45.181978 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.181570    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr9vz\" (UniqueName: \"kubernetes.io/projected/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d-kube-api-access-nr9vz\") pod \"payload-processing-e2e-api-a-5effde-c88b56d97-449fp\" (UID: \"d0cd41f1-8a1d-4d1b-864c-d7bce027e60d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"
Oct 31 03:31:45.181978 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.181673    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5effde-c88b56d97-449fp\" (UID: \"d0cd41f1-8a1d-4d1b-864c-d7bce027e60d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"
Oct 31 03:31:45.195836 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.195781    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"]
Oct 31 03:31:45.259077 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.259026    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5effde\""
Oct 31 03:31:45.259281 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.259117    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5effde-dockercfg-bzd2k\""
Oct 31 03:31:45.282711 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.282678    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr9vz\" (UniqueName: \"kubernetes.io/projected/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d-kube-api-access-nr9vz\") pod \"payload-processing-e2e-api-a-5effde-c88b56d97-449fp\" (UID: \"d0cd41f1-8a1d-4d1b-864c-d7bce027e60d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"
Oct 31 03:31:45.282938 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.282752    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5effde-c88b56d97-449fp\" (UID: \"d0cd41f1-8a1d-4d1b-864c-d7bce027e60d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"
Oct 31 03:31:45.283729 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.283704    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5effde-c88b56d97-449fp\" (UID: \"d0cd41f1-8a1d-4d1b-864c-d7bce027e60d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"
Oct 31 03:31:45.343985 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.343953    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr9vz\" (UniqueName: \"kubernetes.io/projected/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d-kube-api-access-nr9vz\") pod \"payload-processing-e2e-api-a-5effde-c88b56d97-449fp\" (UID: \"d0cd41f1-8a1d-4d1b-864c-d7bce027e60d\") " pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"
Oct 31 03:31:45.493532 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.493490    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"
Oct 31 03:31:45.866746 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:45.866721    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"]
Oct 31 03:31:45.867914 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:31:45.867887    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0cd41f1_8a1d_4d1b_864c_d7bce027e60d.slice/crio-fcf4261613088f89add3134a63ead69cb5c4a3a4d31616de7b86098d1b45e5ea WatchSource:0}: Error finding container fcf4261613088f89add3134a63ead69cb5c4a3a4d31616de7b86098d1b45e5ea: Status 404 returned error can't find the container with id fcf4261613088f89add3134a63ead69cb5c4a3a4d31616de7b86098d1b45e5ea
Oct 31 03:31:46.135158 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:46.135070    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp" event={"ID":"d0cd41f1-8a1d-4d1b-864c-d7bce027e60d","Type":"ContainerStarted","Data":"9a52b3f27f8449e1a3e395a2bb44f3e32888c14536a1225cda21ef30f4e8a31a"}
Oct 31 03:31:46.135158 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:46.135126    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp" event={"ID":"d0cd41f1-8a1d-4d1b-864c-d7bce027e60d","Type":"ContainerStarted","Data":"fcf4261613088f89add3134a63ead69cb5c4a3a4d31616de7b86098d1b45e5ea"}
Oct 31 03:31:46.135385 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:46.135183    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"
Oct 31 03:31:46.242399 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:46.242347    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp" podStartSLOduration=1.242332451 podStartE2EDuration="1.242332451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:46.241003394 +0000 UTC m=+1459.207336739" watchObservedRunningTime="2026-10-31 03:31:46.242332451 +0000 UTC m=+1459.208665796"
Oct 31 03:31:46.676660 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:46.676356    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:49.827177 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:49.827139    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:52.248398 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:52.248265    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:31:57.154008 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:57.153982    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"
Oct 31 03:31:57.795811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:31:57.793079    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:02.837484 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:02.837440    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:06.985928 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:32:06.985890    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:32:07.730408 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:07.730372    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:07.941071 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:07.941040    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:12.940366 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:12.940317    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:13.227168 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:13.227083    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:18.047513 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:18.047474    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:22.627669 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.627629    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"]
Oct 31 03:32:22.631900 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.631873    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:22.723772 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.723736    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-maas-api-tls\") pod \"maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:22.723772 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.723800    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-metrics-certs\") pod \"maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:22.724063 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.723933    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j595\" (UniqueName: \"kubernetes.io/projected/933891b0-abdd-43d0-a94c-eec10ce176e9-kube-api-access-7j595\") pod \"maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:22.724063 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.723994    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933891b0-abdd-43d0-a94c-eec10ce176e9-openshift-service-ca\") pod \"maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:22.729700 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.729668    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2260d8-serving-cert\""
Oct 31 03:32:22.732919 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.732895    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"]
Oct 31 03:32:22.825139 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.825099    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933891b0-abdd-43d0-a94c-eec10ce176e9-openshift-service-ca\") pod \"maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:22.825312 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.825279    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-maas-api-tls\") pod \"maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:22.825354 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.825305    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-metrics-certs\") pod \"maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:22.825400 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.825353    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j595\" (UniqueName: \"kubernetes.io/projected/933891b0-abdd-43d0-a94c-eec10ce176e9-kube-api-access-7j595\") pod \"maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:22.825466 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:32:22.825438    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-serving-cert: secret "maas-api-e2e-api-b-2260d8-serving-cert" not found
Oct 31 03:32:22.825712 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:32:22.825688    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-maas-api-tls podName:933891b0-abdd-43d0-a94c-eec10ce176e9 nodeName:}" failed. No retries permitted until 2026-10-31 03:32:23.325666435 +0000 UTC m=+1496.291999772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-maas-api-tls") pod "maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh" (UID: "933891b0-abdd-43d0-a94c-eec10ce176e9") : secret "maas-api-e2e-api-b-2260d8-serving-cert" not found
Oct 31 03:32:22.825967 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.825945    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933891b0-abdd-43d0-a94c-eec10ce176e9-openshift-service-ca\") pod \"maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:22.827866 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.827847    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-metrics-certs\") pod \"maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:22.887275 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.887197    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j595\" (UniqueName: \"kubernetes.io/projected/933891b0-abdd-43d0-a94c-eec10ce176e9-kube-api-access-7j595\") pod \"maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:22.963257 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:22.963225    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:23.176390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:23.176292    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:23.332996 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:23.332963    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-maas-api-tls\") pod \"maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:23.335466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:23.335437    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-maas-api-tls\") pod \"maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:23.546451 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:23.546403    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:23.917086 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:23.917055    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"]
Oct 31 03:32:23.917731 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:32:23.917707    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933891b0_abdd_43d0_a94c_eec10ce176e9.slice/crio-674be7a49b51cd9418df829d02f8f70e21c39a4c56e43783a75cd2bf7d783b59 WatchSource:0}: Error finding container 674be7a49b51cd9418df829d02f8f70e21c39a4c56e43783a75cd2bf7d783b59: Status 404 returned error can't find the container with id 674be7a49b51cd9418df829d02f8f70e21c39a4c56e43783a75cd2bf7d783b59
Oct 31 03:32:24.382755 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:24.382719    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh" event={"ID":"933891b0-abdd-43d0-a94c-eec10ce176e9","Type":"ContainerStarted","Data":"674be7a49b51cd9418df829d02f8f70e21c39a4c56e43783a75cd2bf7d783b59"}
Oct 31 03:32:25.390175 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:25.390136    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh" event={"ID":"933891b0-abdd-43d0-a94c-eec10ce176e9","Type":"ContainerStarted","Data":"f98ae2cb033b26537acf737de365377e90f2b44f387131392f704a635f4d26ab"}
Oct 31 03:32:25.390596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:25.390218    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:25.497824 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:25.497749    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh" podStartSLOduration=2.60897547 podStartE2EDuration="3.497733149s" totalImagesPullingTime="888.757679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:25.497080955 +0000 UTC m=+1498.463414300" watchObservedRunningTime="2026-10-31 03:32:25.497733149 +0000 UTC m=+1498.464066493"
Oct 31 03:32:27.928477 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:27.928393    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:28.146268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:28.146233    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:28.492382 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:28.492357    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:32:28.531676 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:28.531648    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:32:31.401980 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:31.401950    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:32:33.082702 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:33.082669    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:36.986454 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:32:36.986408    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:32:38.424253 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:38.424214    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:38.942145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:38.942092    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:41.018276 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:41.018243    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:43.556221 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:43.556185    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:46.533337 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:46.533268    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:46.768971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:46.768923    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:46.982345 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:46.982309    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:48.807234 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:48.807199    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:50.526615 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:50.526580    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:50.858559 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:50.858523    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:51.141033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:51.140928    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:51.350669 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:51.350624    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:52.779457 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:52.779398    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:54.590389 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:54.590306    4900 generic.go:356] "Generic (PLEG): container finished" podID="676e6a62-14d6-42a5-81d7-5b08d33de624" containerID="3b7c2b4735ee0268eef08896dff22ba6591ee85777f245f15af48c168915be90" exitCode=0
Oct 31 03:32:54.590389 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:54.590376    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4" event={"ID":"676e6a62-14d6-42a5-81d7-5b08d33de624","Type":"ContainerDied","Data":"3b7c2b4735ee0268eef08896dff22ba6591ee85777f245f15af48c168915be90"}
Oct 31 03:32:54.590916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:54.590897    4900 scope.go:122] "RemoveContainer" containerID="3b7c2b4735ee0268eef08896dff22ba6591ee85777f245f15af48c168915be90"
Oct 31 03:32:54.592381 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:54.592352    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:32:54.592464 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:54.592420    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:32:54.592464 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:54.592452    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:32:55.599013 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:55.598975    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4" event={"ID":"676e6a62-14d6-42a5-81d7-5b08d33de624","Type":"ContainerStarted","Data":"494a00386c17c3142b2a161310cd5a4134b6625bb7287129816f67249a0de7da"}
Oct 31 03:32:55.730948 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:55.730920    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:56.219457 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:56.219419    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:32:56.219737 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:56.219713    4900 patch_prober.go:29] interesting pod/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 03:32:56.219814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:56.219774    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4" podUID="676e6a62-14d6-42a5-81d7-5b08d33de624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 03:32:56.699655 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:56.699616    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:57.219942 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:57.219903    4900 patch_prober.go:29] interesting pod/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 03:32:57.220121 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:57.219966    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4" podUID="676e6a62-14d6-42a5-81d7-5b08d33de624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 03:32:58.218922 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:58.218890    4900 patch_prober.go:29] interesting pod/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 03:32:58.219315 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:58.218939    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4" podUID="676e6a62-14d6-42a5-81d7-5b08d33de624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 03:32:59.219476 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:59.219435    4900 patch_prober.go:29] interesting pod/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 03:32:59.219922 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:59.219493    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4" podUID="676e6a62-14d6-42a5-81d7-5b08d33de624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 03:32:59.378567 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:59.378530    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:32:59.592083 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:32:59.592050    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:00.219518 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:00.219479    4900 patch_prober.go:29] interesting pod/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 03:33:00.219966 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:00.219544    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4" podUID="676e6a62-14d6-42a5-81d7-5b08d33de624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 03:33:01.223175 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:01.223147    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:33:01.223646 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:01.223611    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:33:01.224402 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:01.224383    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:33:02.059205 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:02.059162    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:03.576005 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:03.575952    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:03.784375 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:03.784324    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:04.415097 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:04.415058    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"]
Oct 31 03:33:04.415308 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:04.415272    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z" podUID="f300bdd4-16f8-4e79-89b4-c1e1249c9d14" containerName="istio-proxy" containerID="cri-o://46f6527821eedb36499cf0ac565adfabe5ccc247425f0672282f3ec78502080d" gracePeriod=30
Oct 31 03:33:06.845704 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:06.845650    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:06.994967 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:33:06.994922    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:33:08.893556 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:08.893522    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:09.684640 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.684571    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:33:09.723814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.723756    4900 generic.go:356] "Generic (PLEG): container finished" podID="f300bdd4-16f8-4e79-89b4-c1e1249c9d14" containerID="46f6527821eedb36499cf0ac565adfabe5ccc247425f0672282f3ec78502080d" exitCode=0
Oct 31 03:33:09.724002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.723817    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z" event={"ID":"f300bdd4-16f8-4e79-89b4-c1e1249c9d14","Type":"ContainerDied","Data":"46f6527821eedb36499cf0ac565adfabe5ccc247425f0672282f3ec78502080d"}
Oct 31 03:33:09.724002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.723831    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"
Oct 31 03:33:09.724002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.723873    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z" event={"ID":"f300bdd4-16f8-4e79-89b4-c1e1249c9d14","Type":"ContainerDied","Data":"df53813021fe9b7a911e3231af64939b2dce25006f78df1afec78169b8de0dfe"}
Oct 31 03:33:09.724002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.723899    4900 scope.go:122] "RemoveContainer" containerID="46f6527821eedb36499cf0ac565adfabe5ccc247425f0672282f3ec78502080d"
Oct 31 03:33:09.733560 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.733541    4900 scope.go:122] "RemoveContainer" containerID="46f6527821eedb36499cf0ac565adfabe5ccc247425f0672282f3ec78502080d"
Oct 31 03:33:09.733880 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:33:09.733850    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f6527821eedb36499cf0ac565adfabe5ccc247425f0672282f3ec78502080d\": container with ID starting with 46f6527821eedb36499cf0ac565adfabe5ccc247425f0672282f3ec78502080d not found: ID does not exist" containerID="46f6527821eedb36499cf0ac565adfabe5ccc247425f0672282f3ec78502080d"
Oct 31 03:33:09.733968 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.733880    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f6527821eedb36499cf0ac565adfabe5ccc247425f0672282f3ec78502080d"} err="failed to get container status \"46f6527821eedb36499cf0ac565adfabe5ccc247425f0672282f3ec78502080d\": rpc error: code = NotFound desc = could not find container \"46f6527821eedb36499cf0ac565adfabe5ccc247425f0672282f3ec78502080d\": container with ID starting with 46f6527821eedb36499cf0ac565adfabe5ccc247425f0672282f3ec78502080d not found: ID does not exist"
Oct 31 03:33:09.802495 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.802456    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-workload-certs\") pod \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") "
Oct 31 03:33:09.802495 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.802496    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-envoy\") pod \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") "
Oct 31 03:33:09.802698 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.802540    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-token\" (UniqueName: \"kubernetes.io/projected/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-token\") pod \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") "
Oct 31 03:33:09.802698 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.802636    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-credential-socket\") pod \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") "
Oct 31 03:33:09.802832 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.802781    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-kube-api-access-mfw4j\" (UniqueName: \"kubernetes.io/projected/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-kube-api-access-mfw4j\") pod \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") "
Oct 31 03:33:09.802910 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.802820    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-workload-certs" pod "f300bdd4-16f8-4e79-89b4-c1e1249c9d14" (UID: "f300bdd4-16f8-4e79-89b4-c1e1249c9d14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.802971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.802928    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-credential-socket" pod "f300bdd4-16f8-4e79-89b4-c1e1249c9d14" (UID: "f300bdd4-16f8-4e79-89b4-c1e1249c9d14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.802971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.802941    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istiod-ca-cert\") pod \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") "
Oct 31 03:33:09.803090 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.803065    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-workload-socket\") pod \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") "
Oct 31 03:33:09.803150 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.803105    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-data\") pod \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") "
Oct 31 03:33:09.803150 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.803139    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-podinfo\") pod \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\" (UID: \"f300bdd4-16f8-4e79-89b4-c1e1249c9d14\") "
Oct 31 03:33:09.803351 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.803330    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-data" pod "f300bdd4-16f8-4e79-89b4-c1e1249c9d14" (UID: "f300bdd4-16f8-4e79-89b4-c1e1249c9d14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.803407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.803343    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-workload-socket" pod "f300bdd4-16f8-4e79-89b4-c1e1249c9d14" (UID: "f300bdd4-16f8-4e79-89b4-c1e1249c9d14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.803407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.803362    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istiod-ca-cert" pod "f300bdd4-16f8-4e79-89b4-c1e1249c9d14" (UID: "f300bdd4-16f8-4e79-89b4-c1e1249c9d14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:09.803912 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.803885    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.804040 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.803914    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.804040 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.803931    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.804040 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.803945    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.804040 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.803958    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.804944 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.804918    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-envoy" pod "f300bdd4-16f8-4e79-89b4-c1e1249c9d14" (UID: "f300bdd4-16f8-4e79-89b4-c1e1249c9d14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:09.805038 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.805018    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-kube-api-access-mfw4j" pod "f300bdd4-16f8-4e79-89b4-c1e1249c9d14" (UID: "f300bdd4-16f8-4e79-89b4-c1e1249c9d14"). InnerVolumeSpecName "kube-api-access-mfw4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:09.805149 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.805124    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-token" pod "f300bdd4-16f8-4e79-89b4-c1e1249c9d14" (UID: "f300bdd4-16f8-4e79-89b4-c1e1249c9d14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:09.805373 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.805354    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-podinfo" pod "f300bdd4-16f8-4e79-89b4-c1e1249c9d14" (UID: "f300bdd4-16f8-4e79-89b4-c1e1249c9d14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:09.905325 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.905291    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.905325 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.905322    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.905325 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.905334    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfw4j\" (UniqueName: \"kubernetes.io/projected/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-kube-api-access-mfw4j\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.905820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:09.905344    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f300bdd4-16f8-4e79-89b4-c1e1249c9d14-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:10.151181 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:10.151147    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"]
Oct 31 03:33:10.197823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:10.197772    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z"]
Oct 31 03:33:10.607937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:10.607892    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:33:10.608124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:10.607971    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-fpt9z" podUID="f300bdd4-16f8-4e79-89b4-c1e1249c9d14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:33:11.153697 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:11.153664    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f300bdd4-16f8-4e79-89b4-c1e1249c9d14" path="/var/lib/kubelet/pods/f300bdd4-16f8-4e79-89b4-c1e1249c9d14/volumes"
Oct 31 03:33:14.147688 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:14.147617    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:17.870810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:17.870748    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:18.172352 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:18.172265    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:19.402400 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:19.402356    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:19.696992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:19.696905    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:20.145689 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:20.145651    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:21.008423 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:21.008386    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:22.523274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:22.523236    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"]
Oct 31 03:33:22.523767 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:22.523486    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv" podUID="0de8555d-e0c1-41de-ab04-5a26e8fc76bf" containerName="istio-proxy" containerID="cri-o://86c8e987c0c22b134f093c7f308e807d314178ba768357c16f380248630b230e" gracePeriod=30
Oct 31 03:33:25.088946 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:25.088901    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:27.595619 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.595579    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:27.789583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.789495    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:33:27.875923 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.875894    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-podinfo\") pod \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") "
Oct 31 03:33:27.876126 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.875991    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-workload-certs\") pod \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") "
Oct 31 03:33:27.876126 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.876021    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-token\" (UniqueName: \"kubernetes.io/projected/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-token\") pod \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") "
Oct 31 03:33:27.876126 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.876064    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-kube-api-access-bzggw\" (UniqueName: \"kubernetes.io/projected/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-kube-api-access-bzggw\") pod \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") "
Oct 31 03:33:27.876312 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.876197    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-envoy\") pod \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") "
Oct 31 03:33:27.876312 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.876222    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-data\") pod \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") "
Oct 31 03:33:27.876312 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.876282    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-credential-socket\") pod \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") "
Oct 31 03:33:27.876312 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.876283    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-workload-certs" pod "0de8555d-e0c1-41de-ab04-5a26e8fc76bf" (UID: "0de8555d-e0c1-41de-ab04-5a26e8fc76bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:27.876312 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.876313    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istiod-ca-cert\") pod \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") "
Oct 31 03:33:27.876573 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.876358    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-workload-socket\") pod \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\" (UID: \"0de8555d-e0c1-41de-ab04-5a26e8fc76bf\") "
Oct 31 03:33:27.877004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.876718    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-credential-socket" pod "0de8555d-e0c1-41de-ab04-5a26e8fc76bf" (UID: "0de8555d-e0c1-41de-ab04-5a26e8fc76bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:27.877004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.876748    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-data" pod "0de8555d-e0c1-41de-ab04-5a26e8fc76bf" (UID: "0de8555d-e0c1-41de-ab04-5a26e8fc76bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:27.877004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.876956    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-workload-socket" pod "0de8555d-e0c1-41de-ab04-5a26e8fc76bf" (UID: "0de8555d-e0c1-41de-ab04-5a26e8fc76bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:27.877191 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.877013    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istiod-ca-cert" pod "0de8555d-e0c1-41de-ab04-5a26e8fc76bf" (UID: "0de8555d-e0c1-41de-ab04-5a26e8fc76bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:27.877191 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.877137    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:27.877191 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.877162    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:27.877191 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.877178    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:27.877343 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.877193    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:27.877343 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.877208    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:27.877973 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.877943    4900 generic.go:356] "Generic (PLEG): container finished" podID="0de8555d-e0c1-41de-ab04-5a26e8fc76bf" containerID="86c8e987c0c22b134f093c7f308e807d314178ba768357c16f380248630b230e" exitCode=0
Oct 31 03:33:27.878085 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.878011    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"
Oct 31 03:33:27.878157 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.878016    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv" event={"ID":"0de8555d-e0c1-41de-ab04-5a26e8fc76bf","Type":"ContainerDied","Data":"86c8e987c0c22b134f093c7f308e807d314178ba768357c16f380248630b230e"}
Oct 31 03:33:27.878235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.878216    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv" event={"ID":"0de8555d-e0c1-41de-ab04-5a26e8fc76bf","Type":"ContainerDied","Data":"e6cfd7687b5e2c314f3f96b1b0d153b39b3f57e58709a93bc7f4f28805bd81ac"}
Oct 31 03:33:27.878298 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.878256    4900 scope.go:122] "RemoveContainer" containerID="86c8e987c0c22b134f093c7f308e807d314178ba768357c16f380248630b230e"
Oct 31 03:33:27.878537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.878506    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-podinfo" pod "0de8555d-e0c1-41de-ab04-5a26e8fc76bf" (UID: "0de8555d-e0c1-41de-ab04-5a26e8fc76bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:33:27.878636 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.878602    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-token" pod "0de8555d-e0c1-41de-ab04-5a26e8fc76bf" (UID: "0de8555d-e0c1-41de-ab04-5a26e8fc76bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:27.879495 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.879470    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-envoy" pod "0de8555d-e0c1-41de-ab04-5a26e8fc76bf" (UID: "0de8555d-e0c1-41de-ab04-5a26e8fc76bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:33:27.879716 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.879692    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-kube-api-access-bzggw" pod "0de8555d-e0c1-41de-ab04-5a26e8fc76bf" (UID: "0de8555d-e0c1-41de-ab04-5a26e8fc76bf"). InnerVolumeSpecName "kube-api-access-bzggw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:27.905796 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.905763    4900 scope.go:122] "RemoveContainer" containerID="86c8e987c0c22b134f093c7f308e807d314178ba768357c16f380248630b230e"
Oct 31 03:33:27.906159 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:33:27.906139    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c8e987c0c22b134f093c7f308e807d314178ba768357c16f380248630b230e\": container with ID starting with 86c8e987c0c22b134f093c7f308e807d314178ba768357c16f380248630b230e not found: ID does not exist" containerID="86c8e987c0c22b134f093c7f308e807d314178ba768357c16f380248630b230e"
Oct 31 03:33:27.906240 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.906168    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c8e987c0c22b134f093c7f308e807d314178ba768357c16f380248630b230e"} err="failed to get container status \"86c8e987c0c22b134f093c7f308e807d314178ba768357c16f380248630b230e\": rpc error: code = NotFound desc = could not find container \"86c8e987c0c22b134f093c7f308e807d314178ba768357c16f380248630b230e\": container with ID starting with 86c8e987c0c22b134f093c7f308e807d314178ba768357c16f380248630b230e not found: ID does not exist"
Oct 31 03:33:27.978430 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.978396    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:27.978430 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.978432    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzggw\" (UniqueName: \"kubernetes.io/projected/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-kube-api-access-bzggw\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:27.978656 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.978447    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:27.978656 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:27.978459    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de8555d-e0c1-41de-ab04-5a26e8fc76bf-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:28.298284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:28.298240    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"]
Oct 31 03:33:28.305006 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:28.304961    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-qvwfv"]
Oct 31 03:33:29.147819 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:29.147770    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:29.152547 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:29.152517    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de8555d-e0c1-41de-ab04-5a26e8fc76bf" path="/var/lib/kubelet/pods/0de8555d-e0c1-41de-ab04-5a26e8fc76bf/volumes"
Oct 31 03:33:30.013478 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:30.013443    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:30.427674 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:30.427584    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:30.904470 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:30.904401    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:31.125753 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:31.125702    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:32.407213 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:32.407185    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:33.394950 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:33.394906    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:34.877849 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:34.877781    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:35.660372 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:35.660332    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:36.292348 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:36.292292    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:36.551639 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:36.551542    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:37.057064 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:33:37.056989    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:33:37.582327 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:37.582293    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:37.844016 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:37.837412    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:38.224038 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:38.223939    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:39.436287 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:39.436231    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:39.652275 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:39.652224    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:40.641848 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:40.641816    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:40.848839 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:40.848806    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:41.482671 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:41.482639    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:41.986243 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:41.986178    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:43.339660 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:43.339629    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:43.630521 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:43.630440    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:43.895609 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:43.895517    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:44.129199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:44.129169    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:44.387106 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:44.387073    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:45.210527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:45.210496    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:47.173032 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:47.172994    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:47.490513 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:47.490458    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:49.302656 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:49.302588    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:50.863443 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:50.863397    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:51.098973 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:51.098941    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:54.255420 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:54.255383    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:33:59.096182 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:33:59.096147    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:03.522699 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:03.522653    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:05.388095 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:05.388038    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:06.989696 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:34:06.989649    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:34:09.035434 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:09.035398    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:09.324831 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:09.324708    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:10.559254 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:10.559214    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:10.761182 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:10.761144    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:11.147360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:11.147289    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:12.100355 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:12.100319    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:14.567108 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:14.567057    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:16.107474 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:16.107429    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:17.877263 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:17.877205    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:18.114509 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:18.114476    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:18.925470 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:18.925434    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:21.996017 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:21.995984    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:24.242528 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:24.242492    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:24.488737 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:24.488699    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:24.857036 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:24.856986    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:25.158626 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:25.158505    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:27.425189 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:27.425155    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:29.002145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:29.002097    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:29.405471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:29.405376    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:29.666195 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:29.666110    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:29.906986 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:29.906949    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:32.728477 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:32.728446    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:32.936981 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:32.936955    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:33.182516 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:33.182432    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:34.986195 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:34.986162    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:35.217445 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:35.217404    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:36.986968 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:34:36.986926    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:34:37.571961 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:37.571923    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"]
Oct 31 03:34:37.572222 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:37.572184    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" podUID="9b030d08-9d78-4c66-8729-243e64177d52" containerName="istio-proxy" containerID="cri-o://231e5d0ad082c4ac5e352726fd6503fd897abe7f680c522c6929df488597c96d" gracePeriod=30
Oct 31 03:34:38.605665 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:38.605631    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:39.919770 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:39.919682    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:34:39.919770 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:39.919749    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" podUID="9b030d08-9d78-4c66-8729-243e64177d52" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:34:40.110652 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:40.110593    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:40.361174 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:40.361140    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:42.844695 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.844639    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:34:42.881713 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.881682    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-istio-envoy\") pod \"9b030d08-9d78-4c66-8729-243e64177d52\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") "
Oct 31 03:34:42.881882 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.881772    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-workload-certs\") pod \"9b030d08-9d78-4c66-8729-243e64177d52\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") "
Oct 31 03:34:42.881882 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.881827    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-credential-socket\") pod \"9b030d08-9d78-4c66-8729-243e64177d52\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") "
Oct 31 03:34:42.881967 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.881903    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b030d08-9d78-4c66-8729-243e64177d52-kube-api-access-hwl9r\" (UniqueName: \"kubernetes.io/projected/9b030d08-9d78-4c66-8729-243e64177d52-kube-api-access-hwl9r\") pod \"9b030d08-9d78-4c66-8729-243e64177d52\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") "
Oct 31 03:34:42.881967 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.881953    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-istio-data\") pod \"9b030d08-9d78-4c66-8729-243e64177d52\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") "
Oct 31 03:34:42.882074 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.881977    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b030d08-9d78-4c66-8729-243e64177d52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b030d08-9d78-4c66-8729-243e64177d52-istio-podinfo\") pod \"9b030d08-9d78-4c66-8729-243e64177d52\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") "
Oct 31 03:34:42.882074 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.882015    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b030d08-9d78-4c66-8729-243e64177d52-istio-token\" (UniqueName: \"kubernetes.io/projected/9b030d08-9d78-4c66-8729-243e64177d52-istio-token\") pod \"9b030d08-9d78-4c66-8729-243e64177d52\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") "
Oct 31 03:34:42.882178 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.882105    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-workload-socket\") pod \"9b030d08-9d78-4c66-8729-243e64177d52\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") "
Oct 31 03:34:42.882178 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.882085    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-workload-certs" pod "9b030d08-9d78-4c66-8729-243e64177d52" (UID: "9b030d08-9d78-4c66-8729-243e64177d52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:42.882178 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.882170    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-credential-socket" pod "9b030d08-9d78-4c66-8729-243e64177d52" (UID: "9b030d08-9d78-4c66-8729-243e64177d52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:42.882334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.882245    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b030d08-9d78-4c66-8729-243e64177d52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b030d08-9d78-4c66-8729-243e64177d52-istiod-ca-cert\") pod \"9b030d08-9d78-4c66-8729-243e64177d52\" (UID: \"9b030d08-9d78-4c66-8729-243e64177d52\") "
Oct 31 03:34:42.882334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.882250    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-istio-data" pod "9b030d08-9d78-4c66-8729-243e64177d52" (UID: "9b030d08-9d78-4c66-8729-243e64177d52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:42.882434 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.882330    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-workload-socket" pod "9b030d08-9d78-4c66-8729-243e64177d52" (UID: "9b030d08-9d78-4c66-8729-243e64177d52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:42.882709 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.882682    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b030d08-9d78-4c66-8729-243e64177d52-istiod-ca-cert" pod "9b030d08-9d78-4c66-8729-243e64177d52" (UID: "9b030d08-9d78-4c66-8729-243e64177d52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:34:42.882903 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.882880    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b030d08-9d78-4c66-8729-243e64177d52-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:42.882988 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.882908    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:42.882988 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.882926    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:42.882988 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.882941    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:42.882988 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.882956    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:42.884191 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.884165    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-istio-envoy" pod "9b030d08-9d78-4c66-8729-243e64177d52" (UID: "9b030d08-9d78-4c66-8729-243e64177d52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:34:42.884298 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.884204    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b030d08-9d78-4c66-8729-243e64177d52-istio-token" pod "9b030d08-9d78-4c66-8729-243e64177d52" (UID: "9b030d08-9d78-4c66-8729-243e64177d52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:42.884298 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.884274    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b030d08-9d78-4c66-8729-243e64177d52-istio-podinfo" pod "9b030d08-9d78-4c66-8729-243e64177d52" (UID: "9b030d08-9d78-4c66-8729-243e64177d52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:34:42.884871 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.884844    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b030d08-9d78-4c66-8729-243e64177d52-kube-api-access-hwl9r" pod "9b030d08-9d78-4c66-8729-243e64177d52" (UID: "9b030d08-9d78-4c66-8729-243e64177d52"). InnerVolumeSpecName "kube-api-access-hwl9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:42.984177 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.984143    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b030d08-9d78-4c66-8729-243e64177d52-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:42.984177 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.984173    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwl9r\" (UniqueName: \"kubernetes.io/projected/9b030d08-9d78-4c66-8729-243e64177d52-kube-api-access-hwl9r\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:42.984177 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.984184    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b030d08-9d78-4c66-8729-243e64177d52-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:42.984416 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:42.984193    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b030d08-9d78-4c66-8729-243e64177d52-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:43.499345 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:43.499312    4900 generic.go:356] "Generic (PLEG): container finished" podID="9b030d08-9d78-4c66-8729-243e64177d52" containerID="231e5d0ad082c4ac5e352726fd6503fd897abe7f680c522c6929df488597c96d" exitCode=0
Oct 31 03:34:43.499524 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:43.499357    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" event={"ID":"9b030d08-9d78-4c66-8729-243e64177d52","Type":"ContainerDied","Data":"231e5d0ad082c4ac5e352726fd6503fd897abe7f680c522c6929df488597c96d"}
Oct 31 03:34:43.499524 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:43.499376    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"
Oct 31 03:34:43.499524 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:43.499432    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr" event={"ID":"9b030d08-9d78-4c66-8729-243e64177d52","Type":"ContainerDied","Data":"5b5774dc109adfd9de9482e685ba730f0bb5d182abd939aac4ea5665143e14d3"}
Oct 31 03:34:43.499524 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:43.499458    4900 scope.go:122] "RemoveContainer" containerID="231e5d0ad082c4ac5e352726fd6503fd897abe7f680c522c6929df488597c96d"
Oct 31 03:34:43.509002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:43.508983    4900 scope.go:122] "RemoveContainer" containerID="9b0a944a28df52f75b3d7b0d87d896d03519463561f8948d6f52023db2e66149"
Oct 31 03:34:43.518611 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:43.518588    4900 scope.go:122] "RemoveContainer" containerID="231e5d0ad082c4ac5e352726fd6503fd897abe7f680c522c6929df488597c96d"
Oct 31 03:34:43.519000 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:34:43.518979    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231e5d0ad082c4ac5e352726fd6503fd897abe7f680c522c6929df488597c96d\": container with ID starting with 231e5d0ad082c4ac5e352726fd6503fd897abe7f680c522c6929df488597c96d not found: ID does not exist" containerID="231e5d0ad082c4ac5e352726fd6503fd897abe7f680c522c6929df488597c96d"
Oct 31 03:34:43.519088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:43.519009    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231e5d0ad082c4ac5e352726fd6503fd897abe7f680c522c6929df488597c96d"} err="failed to get container status \"231e5d0ad082c4ac5e352726fd6503fd897abe7f680c522c6929df488597c96d\": rpc error: code = NotFound desc = could not find container \"231e5d0ad082c4ac5e352726fd6503fd897abe7f680c522c6929df488597c96d\": container with ID starting with 231e5d0ad082c4ac5e352726fd6503fd897abe7f680c522c6929df488597c96d not found: ID does not exist"
Oct 31 03:34:43.519088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:43.519029    4900 scope.go:122] "RemoveContainer" containerID="9b0a944a28df52f75b3d7b0d87d896d03519463561f8948d6f52023db2e66149"
Oct 31 03:34:43.519317 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:34:43.519295    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b0a944a28df52f75b3d7b0d87d896d03519463561f8948d6f52023db2e66149\": container with ID starting with 9b0a944a28df52f75b3d7b0d87d896d03519463561f8948d6f52023db2e66149 not found: ID does not exist" containerID="9b0a944a28df52f75b3d7b0d87d896d03519463561f8948d6f52023db2e66149"
Oct 31 03:34:43.519358 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:43.519327    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b0a944a28df52f75b3d7b0d87d896d03519463561f8948d6f52023db2e66149"} err="failed to get container status \"9b0a944a28df52f75b3d7b0d87d896d03519463561f8948d6f52023db2e66149\": rpc error: code = NotFound desc = could not find container \"9b0a944a28df52f75b3d7b0d87d896d03519463561f8948d6f52023db2e66149\": container with ID starting with 9b0a944a28df52f75b3d7b0d87d896d03519463561f8948d6f52023db2e66149 not found: ID does not exist"
Oct 31 03:34:43.700274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:43.700232    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"]
Oct 31 03:34:43.746289 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:43.746251    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-xqpcr"]
Oct 31 03:34:44.190539 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:44.190509    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:45.153806 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:45.153755    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b030d08-9d78-4c66-8729-243e64177d52" path="/var/lib/kubelet/pods/9b030d08-9d78-4c66-8729-243e64177d52/volumes"
Oct 31 03:34:49.017514 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.017481    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"]
Oct 31 03:34:49.018309 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.018288    4900 cpu_manager.go:414] "containerMap: removing container" podUID="9b030d08-9d78-4c66-8729-243e64177d52" containerName="istio-proxy"
Oct 31 03:34:49.018362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.018311    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b030d08-9d78-4c66-8729-243e64177d52" containerName="istio-proxy"
Oct 31 03:34:49.018362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.018350    4900 cpu_manager.go:414] "containerMap: removing container" podUID="f300bdd4-16f8-4e79-89b4-c1e1249c9d14" containerName="istio-proxy"
Oct 31 03:34:49.018362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.018355    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f300bdd4-16f8-4e79-89b4-c1e1249c9d14" containerName="istio-proxy"
Oct 31 03:34:49.018473 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.018368    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0de8555d-e0c1-41de-ab04-5a26e8fc76bf" containerName="istio-proxy"
Oct 31 03:34:49.018473 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.018374    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de8555d-e0c1-41de-ab04-5a26e8fc76bf" containerName="istio-proxy"
Oct 31 03:34:49.018473 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.018457    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b030d08-9d78-4c66-8729-243e64177d52" containerName="istio-proxy"
Oct 31 03:34:49.018473 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.018467    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de8555d-e0c1-41de-ab04-5a26e8fc76bf" containerName="istio-proxy"
Oct 31 03:34:49.018601 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.018481    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b030d08-9d78-4c66-8729-243e64177d52" containerName="istio-proxy"
Oct 31 03:34:49.018601 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.018488    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="f300bdd4-16f8-4e79-89b4-c1e1249c9d14" containerName="istio-proxy"
Oct 31 03:34:49.021957 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.021929    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.071458 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.071423    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6b3a29ec-openshift-default-dockercfg-4ddsz\""
Oct 31 03:34:49.122419 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.122374    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"]
Oct 31 03:34:49.174701 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.174664    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.174883 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.174716    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.174953 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.174923    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.174995 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.174977    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.175185 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.175167    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.175226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.175216    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.175304 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.175284    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpbgd\" (UniqueName: \"kubernetes.io/projected/186cdfd1-f984-4147-8bea-e13f3e50f892-kube-api-access-cpbgd\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.175379 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.175364    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186cdfd1-f984-4147-8bea-e13f3e50f892-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.175424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.175398    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.277251 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.277156    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.277251 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.277207    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.277494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.277356    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.277494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.277391    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.277494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.277418    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpbgd\" (UniqueName: \"kubernetes.io/projected/186cdfd1-f984-4147-8bea-e13f3e50f892-kube-api-access-cpbgd\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.277494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.277482    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186cdfd1-f984-4147-8bea-e13f3e50f892-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.277705 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.277513    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.277705 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.277571    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.277705 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.277604    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.277705 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.277635    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.277941 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.277763    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.278085 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.278058    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.278206 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.278116    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.278312 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.278290    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186cdfd1-f984-4147-8bea-e13f3e50f892-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.279987 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.279965    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.280171 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.280155    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.332519 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.332481    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.332674 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.332572    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpbgd\" (UniqueName: \"kubernetes.io/projected/186cdfd1-f984-4147-8bea-e13f3e50f892-kube-api-access-cpbgd\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.336445 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.336425    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:49.758894 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.758865    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"]
Oct 31 03:34:49.761052 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:34:49.761018    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186cdfd1_f984_4147_8bea_e13f3e50f892.slice/crio-70190f2632af4cea25ac93400406c8189cbca23dfb9be867b769f94475376c25 WatchSource:0}: Error finding container 70190f2632af4cea25ac93400406c8189cbca23dfb9be867b769f94475376c25: Status 404 returned error can't find the container with id 70190f2632af4cea25ac93400406c8189cbca23dfb9be867b769f94475376c25
Oct 31 03:34:49.765300 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.765265    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:34:49.765407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.765347    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:34:49.765407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.765389    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:34:49.785021 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:49.784984    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:50.563694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:50.563658    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh" event={"ID":"186cdfd1-f984-4147-8bea-e13f3e50f892","Type":"ContainerStarted","Data":"8a27d5002531779c074265e3f5e6886f2b6e885cebac6604c8181922c346e23a"}
Oct 31 03:34:50.564133 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:50.563718    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh" event={"ID":"186cdfd1-f984-4147-8bea-e13f3e50f892","Type":"ContainerStarted","Data":"70190f2632af4cea25ac93400406c8189cbca23dfb9be867b769f94475376c25"}
Oct 31 03:34:50.720366 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:50.720297    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh" podStartSLOduration=2.720280026 podStartE2EDuration="2.720280026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:50.719011563 +0000 UTC m=+1643.685344908" watchObservedRunningTime="2026-10-31 03:34:50.720280026 +0000 UTC m=+1643.686613370"
Oct 31 03:34:51.260162 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:51.260124    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:51.349814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:51.344169    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:51.358965 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:51.358925    4900 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 03:34:51.359156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:51.359014    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh" podUID="186cdfd1-f984-4147-8bea-e13f3e50f892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 03:34:51.487977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:51.487943    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:52.355877 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:52.353179    4900 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 03:34:52.355877 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:52.353250    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh" podUID="186cdfd1-f984-4147-8bea-e13f3e50f892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 03:34:53.340221 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:53.340161    4900 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 03:34:53.340535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:53.340507    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh" podUID="186cdfd1-f984-4147-8bea-e13f3e50f892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 03:34:54.341030 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:54.339184    4900 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 03:34:54.341030 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:54.339266    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh" podUID="186cdfd1-f984-4147-8bea-e13f3e50f892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 03:34:55.071694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:55.071659    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:34:55.353868 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:55.353204    4900 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body=
Oct 31 03:34:55.353868 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:55.353282    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh" podUID="186cdfd1-f984-4147-8bea-e13f3e50f892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused"
Oct 31 03:34:56.342313 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:56.342280    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:56.342586 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:56.342566    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:56.343229 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:56.343209    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:34:58.381250 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:34:58.381209    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:00.089240 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:00.089092    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:03.414892 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:03.414860    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:04.144019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.143989    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"]
Oct 31 03:35:04.144593 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.144580    4900 cpu_manager.go:414] "containerMap: removing container" podUID="9b030d08-9d78-4c66-8729-243e64177d52" containerName="istio-proxy"
Oct 31 03:35:04.144637 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.144595    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b030d08-9d78-4c66-8729-243e64177d52" containerName="istio-proxy"
Oct 31 03:35:04.148158 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.148138    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.204500 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.204468    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.204712 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.204515    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.204712 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.204538    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0397d82-9c9b-443f-b763-95d71d5adac9-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.204830 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.204746    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.204830 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.204809    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.204917 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.204891    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktf2r\" (UniqueName: \"kubernetes.io/projected/b0397d82-9c9b-443f-b763-95d71d5adac9-kube-api-access-ktf2r\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.204994 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.204976    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.205080 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.205068    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.205117 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.205107    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.239227 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.239196    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"]
Oct 31 03:35:04.296943 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.296908    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"]
Oct 31 03:35:04.297138 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.297118    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh" podUID="933891b0-abdd-43d0-a94c-eec10ce176e9" containerName="maas-api" containerID="cri-o://f98ae2cb033b26537acf737de365377e90f2b44f387131392f704a635f4d26ab" gracePeriod=30
Oct 31 03:35:04.306367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.306338    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.306509 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.306406    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.306509 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.306441    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktf2r\" (UniqueName: \"kubernetes.io/projected/b0397d82-9c9b-443f-b763-95d71d5adac9-kube-api-access-ktf2r\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.306748 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.306719    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.306874 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.306822    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-workload-certs\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.306874 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.306856    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.306992 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.306923    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.307051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.306985    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.307051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.307044    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.307233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.306860    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-credential-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.307382 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.307269    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0397d82-9c9b-443f-b763-95d71d5adac9-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.307382 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.307299    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-workload-socket\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.307783 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.307760    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-data\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.308313 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.308288    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0397d82-9c9b-443f-b763-95d71d5adac9-istiod-ca-cert\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.309418 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.309392    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-envoy\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.309674 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.309655    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-podinfo\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.363213 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.363176    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-token\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.365757 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.365715    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktf2r\" (UniqueName: \"kubernetes.io/projected/b0397d82-9c9b-443f-b763-95d71d5adac9-kube-api-access-ktf2r\") pod \"e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") " pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.462996 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.462959    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:04.502073 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.502043    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:04.604640 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.604614    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:35:04.647117 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.647077    4900 generic.go:356] "Generic (PLEG): container finished" podID="933891b0-abdd-43d0-a94c-eec10ce176e9" containerID="f98ae2cb033b26537acf737de365377e90f2b44f387131392f704a635f4d26ab" exitCode=0
Oct 31 03:35:04.647284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.647120    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"
Oct 31 03:35:04.647284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.647153    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh" event={"ID":"933891b0-abdd-43d0-a94c-eec10ce176e9","Type":"ContainerDied","Data":"f98ae2cb033b26537acf737de365377e90f2b44f387131392f704a635f4d26ab"}
Oct 31 03:35:04.647284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.647206    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh" event={"ID":"933891b0-abdd-43d0-a94c-eec10ce176e9","Type":"ContainerDied","Data":"674be7a49b51cd9418df829d02f8f70e21c39a4c56e43783a75cd2bf7d783b59"}
Oct 31 03:35:04.647284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.647239    4900 scope.go:122] "RemoveContainer" containerID="f98ae2cb033b26537acf737de365377e90f2b44f387131392f704a635f4d26ab"
Oct 31 03:35:04.659934 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.659913    4900 scope.go:122] "RemoveContainer" containerID="f98ae2cb033b26537acf737de365377e90f2b44f387131392f704a635f4d26ab"
Oct 31 03:35:04.660270 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:35:04.660247    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98ae2cb033b26537acf737de365377e90f2b44f387131392f704a635f4d26ab\": container with ID starting with f98ae2cb033b26537acf737de365377e90f2b44f387131392f704a635f4d26ab not found: ID does not exist" containerID="f98ae2cb033b26537acf737de365377e90f2b44f387131392f704a635f4d26ab"
Oct 31 03:35:04.660373 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.660276    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98ae2cb033b26537acf737de365377e90f2b44f387131392f704a635f4d26ab"} err="failed to get container status \"f98ae2cb033b26537acf737de365377e90f2b44f387131392f704a635f4d26ab\": rpc error: code = NotFound desc = could not find container \"f98ae2cb033b26537acf737de365377e90f2b44f387131392f704a635f4d26ab\": container with ID starting with f98ae2cb033b26537acf737de365377e90f2b44f387131392f704a635f4d26ab not found: ID does not exist"
Oct 31 03:35:04.741898 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.728858    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-metrics-certs\") pod \"933891b0-abdd-43d0-a94c-eec10ce176e9\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") "
Oct 31 03:35:04.741898 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.729178    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933891b0-abdd-43d0-a94c-eec10ce176e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933891b0-abdd-43d0-a94c-eec10ce176e9-openshift-service-ca\") pod \"933891b0-abdd-43d0-a94c-eec10ce176e9\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") "
Oct 31 03:35:04.741898 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.729318    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-maas-api-tls\") pod \"933891b0-abdd-43d0-a94c-eec10ce176e9\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") "
Oct 31 03:35:04.741898 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.729480    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933891b0-abdd-43d0-a94c-eec10ce176e9-kube-api-access-7j595\" (UniqueName: \"kubernetes.io/projected/933891b0-abdd-43d0-a94c-eec10ce176e9-kube-api-access-7j595\") pod \"933891b0-abdd-43d0-a94c-eec10ce176e9\" (UID: \"933891b0-abdd-43d0-a94c-eec10ce176e9\") "
Oct 31 03:35:04.741898 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.731251    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933891b0-abdd-43d0-a94c-eec10ce176e9-openshift-service-ca" pod "933891b0-abdd-43d0-a94c-eec10ce176e9" (UID: "933891b0-abdd-43d0-a94c-eec10ce176e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:35:04.741898 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.737615    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-maas-api-tls" pod "933891b0-abdd-43d0-a94c-eec10ce176e9" (UID: "933891b0-abdd-43d0-a94c-eec10ce176e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:35:04.768737 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.768567    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933891b0-abdd-43d0-a94c-eec10ce176e9-kube-api-access-7j595" pod "933891b0-abdd-43d0-a94c-eec10ce176e9" (UID: "933891b0-abdd-43d0-a94c-eec10ce176e9"). InnerVolumeSpecName "kube-api-access-7j595". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:35:04.768965 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.768936    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-metrics-certs" pod "933891b0-abdd-43d0-a94c-eec10ce176e9" (UID: "933891b0-abdd-43d0-a94c-eec10ce176e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:35:04.831620 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.831583    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:04.831620 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.831616    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933891b0-abdd-43d0-a94c-eec10ce176e9-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:04.831854 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.831631    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933891b0-abdd-43d0-a94c-eec10ce176e9-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:04.831854 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.831644    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j595\" (UniqueName: \"kubernetes.io/projected/933891b0-abdd-43d0-a94c-eec10ce176e9-kube-api-access-7j595\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:04.922911 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.922876    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"]
Oct 31 03:35:04.923132 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:35:04.923106    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0397d82_9c9b_443f_b763_95d71d5adac9.slice/crio-7c60da7291ed431af25001caa0b78867a35319813cadb712d04180a65c2dc4c2 WatchSource:0}: Error finding container 7c60da7291ed431af25001caa0b78867a35319813cadb712d04180a65c2dc4c2: Status 404 returned error can't find the container with id 7c60da7291ed431af25001caa0b78867a35319813cadb712d04180a65c2dc4c2
Oct 31 03:35:04.926071 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.926037    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:35:04.926150 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.926108    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:35:04.926150 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:04.926137    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:35:05.064118 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:05.064082    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"]
Oct 31 03:35:05.068440 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:05.068406    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2260d8-68885b44f8-lm6hh"]
Oct 31 03:35:05.154510 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:05.154472    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933891b0-abdd-43d0-a94c-eec10ce176e9" path="/var/lib/kubelet/pods/933891b0-abdd-43d0-a94c-eec10ce176e9/volumes"
Oct 31 03:35:05.663810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:05.663697    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b" event={"ID":"b0397d82-9c9b-443f-b763-95d71d5adac9","Type":"ContainerStarted","Data":"6c214fed12307f1e6eef206162a023161df5f51957fba56db9318a570ffc67d7"}
Oct 31 03:35:05.663810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:05.663754    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b" event={"ID":"b0397d82-9c9b-443f-b763-95d71d5adac9","Type":"ContainerStarted","Data":"7c60da7291ed431af25001caa0b78867a35319813cadb712d04180a65c2dc4c2"}
Oct 31 03:35:05.775923 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:05.775865    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b" podStartSLOduration=1.775850409 podStartE2EDuration="1.775850409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:05.773021223 +0000 UTC m=+1658.739354572" watchObservedRunningTime="2026-10-31 03:35:05.775850409 +0000 UTC m=+1658.742183753"
Oct 31 03:35:06.469200 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:06.469164    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:06.483272 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:06.483235    4900 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:35:06.483574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:06.483540    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b" podUID="b0397d82-9c9b-443f-b763-95d71d5adac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 03:35:06.498675 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:06.498619    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:06.988553 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:35:06.988520    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:35:07.463499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:07.463411    4900 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:35:07.463499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:07.463465    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b" podUID="b0397d82-9c9b-443f-b763-95d71d5adac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 03:35:08.177484 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:08.177444    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:08.408817 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:08.408755    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:08.464438 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:08.464342    4900 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:35:08.464438 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:08.464409    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b" podUID="b0397d82-9c9b-443f-b763-95d71d5adac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 03:35:09.464127 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:09.464085    4900 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:35:09.464538 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:09.464143    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b" podUID="b0397d82-9c9b-443f-b763-95d71d5adac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 03:35:09.745765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:09.745728    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:10.231424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:10.231344    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:10.466337 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:10.466276    4900 patch_prober.go:29] interesting pod/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Oct 31 03:35:10.467083 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:10.466345    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b" podUID="b0397d82-9c9b-443f-b763-95d71d5adac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Oct 31 03:35:11.456085 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:11.456053    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:11.493354 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:11.493311    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:11.493953 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:11.493930    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:11.494752 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:11.494726    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:35:12.023113 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:12.023081    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:12.258809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:12.258544    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:12.512209 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:12.512179    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:16.477987 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:16.477943    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:16.719223 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:16.719191    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:17.536115 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:17.536081    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:19.870311 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:19.870275    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:20.138374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:20.138288    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:22.970667 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:22.970619    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:23.471437 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:23.471396    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:23.689662 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:23.689631    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:28.412651 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:28.412562    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:28.644142 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:28.644103    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:33.510022 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:33.509989    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:36.992104 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:35:36.992070    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:35:38.612072 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:38.612037    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:39.569124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:39.569096    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:40.386108 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:40.386073    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:41.129543 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:41.129510    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:41.996779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:41.996744    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:44.027629 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:44.027596    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:44.292087 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:44.292000    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:44.514605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:44.514558    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:46.748222 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:46.748177    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:47.698895 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:47.698856    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:48.353503 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:48.353472    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"]
Oct 31 03:35:48.814638 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:48.814602    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:49.065578 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:49.065493    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:49.670189 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:49.670148    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:51.654658 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:51.654626    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:53.154174 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.154142    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:35:53.353860 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.353823    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp" podUID="d0cd41f1-8a1d-4d1b-864c-d7bce027e60d" containerName="payload-processing" containerID="cri-o://9a52b3f27f8449e1a3e395a2bb44f3e32888c14536a1225cda21ef30f4e8a31a" gracePeriod=25
Oct 31 03:35:53.613058 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.613030    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"
Oct 31 03:35:53.755948 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.755914    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d-kube-api-access-nr9vz\" (UniqueName: \"kubernetes.io/projected/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d-kube-api-access-nr9vz\") pod \"d0cd41f1-8a1d-4d1b-864c-d7bce027e60d\" (UID: \"d0cd41f1-8a1d-4d1b-864c-d7bce027e60d\") "
Oct 31 03:35:53.756145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.755970    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d-plugins-config-volume\") pod \"d0cd41f1-8a1d-4d1b-864c-d7bce027e60d\" (UID: \"d0cd41f1-8a1d-4d1b-864c-d7bce027e60d\") "
Oct 31 03:35:53.756416 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.756381    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d-plugins-config-volume" pod "d0cd41f1-8a1d-4d1b-864c-d7bce027e60d" (UID: "d0cd41f1-8a1d-4d1b-864c-d7bce027e60d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:35:53.756687 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.756666    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:53.758235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.758212    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d-kube-api-access-nr9vz" pod "d0cd41f1-8a1d-4d1b-864c-d7bce027e60d" (UID: "d0cd41f1-8a1d-4d1b-864c-d7bce027e60d"). InnerVolumeSpecName "kube-api-access-nr9vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:35:53.858082 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.858042    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr9vz\" (UniqueName: \"kubernetes.io/projected/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d-kube-api-access-nr9vz\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:53.967842 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.967809    4900 generic.go:356] "Generic (PLEG): container finished" podID="d0cd41f1-8a1d-4d1b-864c-d7bce027e60d" containerID="9a52b3f27f8449e1a3e395a2bb44f3e32888c14536a1225cda21ef30f4e8a31a" exitCode=0
Oct 31 03:35:53.968030 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.967922    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"
Oct 31 03:35:53.968030 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.967975    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp" event={"ID":"d0cd41f1-8a1d-4d1b-864c-d7bce027e60d","Type":"ContainerDied","Data":"9a52b3f27f8449e1a3e395a2bb44f3e32888c14536a1225cda21ef30f4e8a31a"}
Oct 31 03:35:53.968115 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.968061    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp" event={"ID":"d0cd41f1-8a1d-4d1b-864c-d7bce027e60d","Type":"ContainerDied","Data":"fcf4261613088f89add3134a63ead69cb5c4a3a4d31616de7b86098d1b45e5ea"}
Oct 31 03:35:53.968115 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.968086    4900 scope.go:122] "RemoveContainer" containerID="9a52b3f27f8449e1a3e395a2bb44f3e32888c14536a1225cda21ef30f4e8a31a"
Oct 31 03:35:53.978946 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.978920    4900 scope.go:122] "RemoveContainer" containerID="9a52b3f27f8449e1a3e395a2bb44f3e32888c14536a1225cda21ef30f4e8a31a"
Oct 31 03:35:53.979224 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:35:53.979207    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a52b3f27f8449e1a3e395a2bb44f3e32888c14536a1225cda21ef30f4e8a31a\": container with ID starting with 9a52b3f27f8449e1a3e395a2bb44f3e32888c14536a1225cda21ef30f4e8a31a not found: ID does not exist" containerID="9a52b3f27f8449e1a3e395a2bb44f3e32888c14536a1225cda21ef30f4e8a31a"
Oct 31 03:35:53.979290 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:53.979231    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a52b3f27f8449e1a3e395a2bb44f3e32888c14536a1225cda21ef30f4e8a31a"} err="failed to get container status \"9a52b3f27f8449e1a3e395a2bb44f3e32888c14536a1225cda21ef30f4e8a31a\": rpc error: code = NotFound desc = could not find container \"9a52b3f27f8449e1a3e395a2bb44f3e32888c14536a1225cda21ef30f4e8a31a\": container with ID starting with 9a52b3f27f8449e1a3e395a2bb44f3e32888c14536a1225cda21ef30f4e8a31a not found: ID does not exist"
Oct 31 03:35:54.120937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:54.120857    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"]
Oct 31 03:35:54.126716 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:54.126689    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5effde-c88b56d97-449fp"]
Oct 31 03:35:55.153773 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:55.153739    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0cd41f1-8a1d-4d1b-864c-d7bce027e60d" path="/var/lib/kubelet/pods/d0cd41f1-8a1d-4d1b-864c-d7bce027e60d/volumes"
Oct 31 03:35:55.452333 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:35:55.452241    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:00.222076 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.222039    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"]
Oct 31 03:36:00.222626 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.222612    4900 cpu_manager.go:414] "containerMap: removing container" podUID="933891b0-abdd-43d0-a94c-eec10ce176e9" containerName="maas-api"
Oct 31 03:36:00.222674 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.222627    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933891b0-abdd-43d0-a94c-eec10ce176e9" containerName="maas-api"
Oct 31 03:36:00.222674 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.222642    4900 cpu_manager.go:414] "containerMap: removing container" podUID="d0cd41f1-8a1d-4d1b-864c-d7bce027e60d" containerName="payload-processing"
Oct 31 03:36:00.222674 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.222647    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0cd41f1-8a1d-4d1b-864c-d7bce027e60d" containerName="payload-processing"
Oct 31 03:36:00.222769 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.222755    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="933891b0-abdd-43d0-a94c-eec10ce176e9" containerName="maas-api"
Oct 31 03:36:00.222769 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.222765    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0cd41f1-8a1d-4d1b-864c-d7bce027e60d" containerName="payload-processing"
Oct 31 03:36:00.227480 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.227460    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.244911 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.244874    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-workload-certs\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.245105 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.245085    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-workload-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.245173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.245161    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-data\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.245274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.245258    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-token\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.245310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.245286    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istiod-ca-cert\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.245357 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.245339    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-podinfo\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.245427 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.245415    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-envoy\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.245549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.245534    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-credential-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.245761 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.245742    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvvfm\" (UniqueName: \"kubernetes.io/projected/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-kube-api-access-qvvfm\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.275816 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.275780    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e1aad0-openshift-default-dockercfg-tlhkf\""
Oct 31 03:36:00.326657 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.326623    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"]
Oct 31 03:36:00.347194 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.347155    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-credential-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.347389 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.347223    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvvfm\" (UniqueName: \"kubernetes.io/projected/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-kube-api-access-qvvfm\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.347389 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.347317    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-workload-certs\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.347389 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.347362    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-workload-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.347579 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.347402    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-data\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.347579 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.347462    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-token\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.347579 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.347493    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istiod-ca-cert\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.347579 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.347528    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-podinfo\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.347799 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.347599    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-envoy\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.347799 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.347673    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-credential-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.347799 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.347691    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-workload-certs\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.347799 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.347772    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-data\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.348083 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.348060    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-workload-socket\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.348416 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.348385    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istiod-ca-cert\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.350581 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.350553    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-envoy\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.351312 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.351291    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-podinfo\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.402284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.402247    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvvfm\" (UniqueName: \"kubernetes.io/projected/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-kube-api-access-qvvfm\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.436355 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.436320    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-token\") pod \"e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") " pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.540662 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.540627    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:00.703677 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.703641    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:00.967761 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.967735    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"]
Oct 31 03:36:00.968842 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:36:00.968814    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7745c0a4_53b5_40f9_9b78_b4d510f0c6c5.slice/crio-c0db1ffdd8c2e6b362175c12a54d240db29217780674a9e3b613bb3e2d1bd610 WatchSource:0}: Error finding container c0db1ffdd8c2e6b362175c12a54d240db29217780674a9e3b613bb3e2d1bd610: Status 404 returned error can't find the container with id c0db1ffdd8c2e6b362175c12a54d240db29217780674a9e3b613bb3e2d1bd610
Oct 31 03:36:00.971546 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.971515    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:36:00.971624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.971579    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:36:00.971624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:00.971610    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:36:01.013916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:01.013887    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr" event={"ID":"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5","Type":"ContainerStarted","Data":"c0db1ffdd8c2e6b362175c12a54d240db29217780674a9e3b613bb3e2d1bd610"}
Oct 31 03:36:02.021407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:02.021366    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr" event={"ID":"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5","Type":"ContainerStarted","Data":"9a3bd92b9ab6c5a37bf09bf98c30825cc0a688802a3c3882fcd77992aa71f175"}
Oct 31 03:36:02.542429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:02.542393    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:02.542818 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:02.542774    4900 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:36:02.542980 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:02.542955    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr" podUID="7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 03:36:03.543273 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:03.543232    4900 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:36:03.543863 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:03.543830    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr" podUID="7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 03:36:04.551820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:04.551236    4900 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:36:04.551820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:04.551303    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr" podUID="7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 03:36:05.558825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:05.553905    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:05.558825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:05.558265    4900 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:36:05.558825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:05.558329    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr" podUID="7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 03:36:05.808324 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:05.808294    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:06.542041 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:06.541165    4900 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 31 03:36:06.542041 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:06.541242    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr" podUID="7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 31 03:36:07.003424 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:36:07.003371    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:36:07.546160 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:07.546125    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:07.546701 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:07.546667    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:07.547454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:07.547436    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:36:07.741752 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:07.741679    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr" podStartSLOduration=7.741664183 podStartE2EDuration="7.741664183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:02.223241192 +0000 UTC m=+1715.189574535" watchObservedRunningTime="2026-10-31 03:36:07.741664183 +0000 UTC m=+1720.707997526"
Oct 31 03:36:08.198421 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:08.198390    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:08.597157 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:08.597128    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:08.890497 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:08.890406    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:10.136047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:10.136010    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:10.562868 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:10.562832    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"]
Oct 31 03:36:10.563108 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:10.563085    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4" podUID="676e6a62-14d6-42a5-81d7-5b08d33de624" containerName="istio-proxy" containerID="cri-o://494a00386c17c3142b2a161310cd5a4134b6625bb7287129816f67249a0de7da" gracePeriod=30
Oct 31 03:36:10.641371 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:10.641337    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:11.416653 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:11.416624    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:13.686803 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:13.686760    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:13.927457 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:13.927415    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:14.152591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:14.152559    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:15.858015 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.857988    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:36:15.901235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.901202    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-credential-socket\") pod \"676e6a62-14d6-42a5-81d7-5b08d33de624\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") "
Oct 31 03:36:15.901378 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.901313    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-workload-socket\") pod \"676e6a62-14d6-42a5-81d7-5b08d33de624\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") "
Oct 31 03:36:15.901378 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.901351    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676e6a62-14d6-42a5-81d7-5b08d33de624-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676e6a62-14d6-42a5-81d7-5b08d33de624-istiod-ca-cert\") pod \"676e6a62-14d6-42a5-81d7-5b08d33de624\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") "
Oct 31 03:36:15.901486 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.901462    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-credential-socket" pod "676e6a62-14d6-42a5-81d7-5b08d33de624" (UID: "676e6a62-14d6-42a5-81d7-5b08d33de624"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:15.901523 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.901469    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-envoy\") pod \"676e6a62-14d6-42a5-81d7-5b08d33de624\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") "
Oct 31 03:36:15.901593 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.901581    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-data\") pod \"676e6a62-14d6-42a5-81d7-5b08d33de624\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") "
Oct 31 03:36:15.901632 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.901608    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676e6a62-14d6-42a5-81d7-5b08d33de624-kube-api-access-tzts9\" (UniqueName: \"kubernetes.io/projected/676e6a62-14d6-42a5-81d7-5b08d33de624-kube-api-access-tzts9\") pod \"676e6a62-14d6-42a5-81d7-5b08d33de624\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") "
Oct 31 03:36:15.901632 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.901618    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-workload-socket" pod "676e6a62-14d6-42a5-81d7-5b08d33de624" (UID: "676e6a62-14d6-42a5-81d7-5b08d33de624"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:15.901744 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.901631    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-podinfo\") pod \"676e6a62-14d6-42a5-81d7-5b08d33de624\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") "
Oct 31 03:36:15.901744 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.901692    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-token\" (UniqueName: \"kubernetes.io/projected/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-token\") pod \"676e6a62-14d6-42a5-81d7-5b08d33de624\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") "
Oct 31 03:36:15.901889 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.901756    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-workload-certs\") pod \"676e6a62-14d6-42a5-81d7-5b08d33de624\" (UID: \"676e6a62-14d6-42a5-81d7-5b08d33de624\") "
Oct 31 03:36:15.901889 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.901839    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-data" pod "676e6a62-14d6-42a5-81d7-5b08d33de624" (UID: "676e6a62-14d6-42a5-81d7-5b08d33de624"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:15.901889 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.901837    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676e6a62-14d6-42a5-81d7-5b08d33de624-istiod-ca-cert" pod "676e6a62-14d6-42a5-81d7-5b08d33de624" (UID: "676e6a62-14d6-42a5-81d7-5b08d33de624"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:36:15.902194 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.902168    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-workload-certs" pod "676e6a62-14d6-42a5-81d7-5b08d33de624" (UID: "676e6a62-14d6-42a5-81d7-5b08d33de624"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:15.902453 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.902431    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:15.902508 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.902463    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:15.902508 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.902479    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:15.902508 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.902495    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:15.902605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.902509    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/676e6a62-14d6-42a5-81d7-5b08d33de624-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:15.903694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.903668    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-podinfo" pod "676e6a62-14d6-42a5-81d7-5b08d33de624" (UID: "676e6a62-14d6-42a5-81d7-5b08d33de624"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:36:15.903825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.903719    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-envoy" pod "676e6a62-14d6-42a5-81d7-5b08d33de624" (UID: "676e6a62-14d6-42a5-81d7-5b08d33de624"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:15.904132 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.904117    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-token" pod "676e6a62-14d6-42a5-81d7-5b08d33de624" (UID: "676e6a62-14d6-42a5-81d7-5b08d33de624"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:15.904196 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:15.904138    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676e6a62-14d6-42a5-81d7-5b08d33de624-kube-api-access-tzts9" pod "676e6a62-14d6-42a5-81d7-5b08d33de624" (UID: "676e6a62-14d6-42a5-81d7-5b08d33de624"). InnerVolumeSpecName "kube-api-access-tzts9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:16.003466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.003434    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:16.003466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.003460    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzts9\" (UniqueName: \"kubernetes.io/projected/676e6a62-14d6-42a5-81d7-5b08d33de624-kube-api-access-tzts9\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:16.003466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.003472    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:16.003691 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.003481    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/676e6a62-14d6-42a5-81d7-5b08d33de624-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:16.151318 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.151191    4900 generic.go:356] "Generic (PLEG): container finished" podID="676e6a62-14d6-42a5-81d7-5b08d33de624" containerID="494a00386c17c3142b2a161310cd5a4134b6625bb7287129816f67249a0de7da" exitCode=0
Oct 31 03:36:16.151318 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.151247    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"
Oct 31 03:36:16.151318 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.151297    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4" event={"ID":"676e6a62-14d6-42a5-81d7-5b08d33de624","Type":"ContainerDied","Data":"494a00386c17c3142b2a161310cd5a4134b6625bb7287129816f67249a0de7da"}
Oct 31 03:36:16.151628 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.151340    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4" event={"ID":"676e6a62-14d6-42a5-81d7-5b08d33de624","Type":"ContainerDied","Data":"152e5b06bae66e53be3c7cd1348996cfbcc68fe7ec5fe3bc93549bc386cea7c9"}
Oct 31 03:36:16.151628 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.151369    4900 scope.go:122] "RemoveContainer" containerID="494a00386c17c3142b2a161310cd5a4134b6625bb7287129816f67249a0de7da"
Oct 31 03:36:16.180514 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.180475    4900 scope.go:122] "RemoveContainer" containerID="3b7c2b4735ee0268eef08896dff22ba6591ee85777f245f15af48c168915be90"
Oct 31 03:36:16.193323 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.193298    4900 scope.go:122] "RemoveContainer" containerID="494a00386c17c3142b2a161310cd5a4134b6625bb7287129816f67249a0de7da"
Oct 31 03:36:16.193638 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:36:16.193612    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494a00386c17c3142b2a161310cd5a4134b6625bb7287129816f67249a0de7da\": container with ID starting with 494a00386c17c3142b2a161310cd5a4134b6625bb7287129816f67249a0de7da not found: ID does not exist" containerID="494a00386c17c3142b2a161310cd5a4134b6625bb7287129816f67249a0de7da"
Oct 31 03:36:16.193699 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.193649    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494a00386c17c3142b2a161310cd5a4134b6625bb7287129816f67249a0de7da"} err="failed to get container status \"494a00386c17c3142b2a161310cd5a4134b6625bb7287129816f67249a0de7da\": rpc error: code = NotFound desc = could not find container \"494a00386c17c3142b2a161310cd5a4134b6625bb7287129816f67249a0de7da\": container with ID starting with 494a00386c17c3142b2a161310cd5a4134b6625bb7287129816f67249a0de7da not found: ID does not exist"
Oct 31 03:36:16.193699 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.193668    4900 scope.go:122] "RemoveContainer" containerID="3b7c2b4735ee0268eef08896dff22ba6591ee85777f245f15af48c168915be90"
Oct 31 03:36:16.194015 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:36:16.193995    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b7c2b4735ee0268eef08896dff22ba6591ee85777f245f15af48c168915be90\": container with ID starting with 3b7c2b4735ee0268eef08896dff22ba6591ee85777f245f15af48c168915be90 not found: ID does not exist" containerID="3b7c2b4735ee0268eef08896dff22ba6591ee85777f245f15af48c168915be90"
Oct 31 03:36:16.194086 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.194024    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b7c2b4735ee0268eef08896dff22ba6591ee85777f245f15af48c168915be90"} err="failed to get container status \"3b7c2b4735ee0268eef08896dff22ba6591ee85777f245f15af48c168915be90\": rpc error: code = NotFound desc = could not find container \"3b7c2b4735ee0268eef08896dff22ba6591ee85777f245f15af48c168915be90\": container with ID starting with 3b7c2b4735ee0268eef08896dff22ba6591ee85777f245f15af48c168915be90 not found: ID does not exist"
Oct 31 03:36:16.306428 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.306392    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"]
Oct 31 03:36:16.311527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.311499    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5effde-openshift-default-78c4445cc5-6l5w4"]
Oct 31 03:36:16.926560 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:16.926526    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:17.160030 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:17.159983    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676e6a62-14d6-42a5-81d7-5b08d33de624" path="/var/lib/kubelet/pods/676e6a62-14d6-42a5-81d7-5b08d33de624/volumes"
Oct 31 03:36:17.189316 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:17.189163    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:17.629469 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:17.629435    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:19.339842 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:19.339764    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:23.580767 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:23.580729    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:24.703039 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:24.703003    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:30.239419 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:30.239378    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:32.276778 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:32.276745    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:33.179598 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:33.179569    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:33.802757 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:33.802724    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:34.197856 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:34.197734    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:34.544831 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:34.544731    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:34.776075 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:34.776040    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:35.185801 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:35.185764    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:36.985760 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:36:36.985719    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:36:37.492853 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:37.492815    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:37.729819 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:37.729758    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:38.231124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:38.231075    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:38.475380 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:38.475342    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:39.546396 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:39.546355    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:39.767602 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:39.767558    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:40.775329 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:40.775297    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:41.270351 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:41.270314    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:41.658744 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:41.658595    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:41.898153 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:41.898124    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:42.261996 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:42.261961    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:42.518181 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:42.518105    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:42.831497 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:42.831424    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:43.178030 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:43.177943    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:43.474826 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:43.474695    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:46.487461 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:46.487428    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:46.723375 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:46.723344    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:46.954634 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:46.954523    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:48.119290 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:48.119255    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:50.385007 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:50.384971    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:50.621429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:50.621381    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:50.860812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:50.859209    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:52.003440 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:52.003413    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:54.800717 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:54.800683    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:57.876101 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:57.876064    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:36:58.084045 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:36:58.083942    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:00.431407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:00.431369    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:05.297513 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:05.297476    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:06.274373 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:06.274335    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"]
Oct 31 03:37:06.274568 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:06.274533    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs" podUID="3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" containerName="istio-proxy" containerID="cri-o://dfd57efd3483a9f84b5d93ccb77da4a41dce0efcf090b30b14311484fe72211a" gracePeriod=30
Oct 31 03:37:06.985161 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:37:06.985126    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:37:10.314230 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:10.314190    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:11.532564 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.532528    4900 generic.go:356] "Generic (PLEG): container finished" podID="3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" containerID="dfd57efd3483a9f84b5d93ccb77da4a41dce0efcf090b30b14311484fe72211a" exitCode=0
Oct 31 03:37:11.532960 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.532596    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs" event={"ID":"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645","Type":"ContainerDied","Data":"dfd57efd3483a9f84b5d93ccb77da4a41dce0efcf090b30b14311484fe72211a"}
Oct 31 03:37:11.532960 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.532664    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs" event={"ID":"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645","Type":"ContainerDied","Data":"db14ae15aa41686e5ff6f2bcfbcc687baea929e7f35fb3b9a1a75b2fa4de4b88"}
Oct 31 03:37:11.532960 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.532693    4900 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db14ae15aa41686e5ff6f2bcfbcc687baea929e7f35fb3b9a1a75b2fa4de4b88"
Oct 31 03:37:11.541563 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.541512    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:37:11.616148 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.616115    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-workload-certs\") pod \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") "
Oct 31 03:37:11.616308 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.616255    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-data\") pod \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") "
Oct 31 03:37:11.616373 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.616312    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istiod-ca-cert\") pod \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") "
Oct 31 03:37:11.616373 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.616340    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-workload-socket\") pod \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") "
Oct 31 03:37:11.616373 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.616363    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-token\" (UniqueName: \"kubernetes.io/projected/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-token\") pod \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") "
Oct 31 03:37:11.616505 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.616406    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-credential-socket\") pod \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") "
Oct 31 03:37:11.616505 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.616401    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-workload-certs" pod "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" (UID: "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:11.616593 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.616505    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-data" pod "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" (UID: "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:11.616593 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.616509    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-podinfo\") pod \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") "
Oct 31 03:37:11.616678 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.616630    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-envoy\") pod \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") "
Oct 31 03:37:11.616678 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.616662    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-kube-api-access-g9b78\" (UniqueName: \"kubernetes.io/projected/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-kube-api-access-g9b78\") pod \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\" (UID: \"3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645\") "
Oct 31 03:37:11.616678 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.616671    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-credential-socket" pod "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" (UID: "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:11.616897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.616689    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-workload-socket" pod "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" (UID: "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:11.616897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.616819    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istiod-ca-cert" pod "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" (UID: "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:11.617244 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.617221    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:11.617244 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.617240    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:11.617429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.617253    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:11.617429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.617267    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:11.617429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.617279    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:11.618759 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.618732    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-kube-api-access-g9b78" pod "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" (UID: "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645"). InnerVolumeSpecName "kube-api-access-g9b78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:11.618885 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.618754    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-envoy" pod "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" (UID: "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:11.618885 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.618826    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-podinfo" pod "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" (UID: "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:11.619012 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.618900    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-token" pod "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" (UID: "3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:11.718317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.718282    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:11.718317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.718310    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:11.718317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.718320    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:11.718644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:11.718332    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9b78\" (UniqueName: \"kubernetes.io/projected/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645-kube-api-access-g9b78\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:12.537734 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:12.537702    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"
Oct 31 03:37:12.738306 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:12.738272    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"]
Oct 31 03:37:12.784469 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:12.784436    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-xpsrs"]
Oct 31 03:37:13.157402 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:13.157366    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" path="/var/lib/kubelet/pods/3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645/volumes"
Oct 31 03:37:14.231395 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:14.231353    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:14.594831 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:14.594708    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:14.800197 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:14.800165    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:15.754029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:15.753996    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:19.424069 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:19.424025    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:19.989735 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:19.989688    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:20.725431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:20.725394    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:22.955425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:22.955392    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:26.582585 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:26.582536    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:27.122972 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:27.122934    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:28.044768 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:28.044725    4900 scope.go:122] "RemoveContainer" containerID="dfd57efd3483a9f84b5d93ccb77da4a41dce0efcf090b30b14311484fe72211a"
Oct 31 03:37:28.435028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:28.434947    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"]
Oct 31 03:37:28.435191 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:28.435145    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm" podUID="88fcbedb-2530-4fc4-b9ec-b897333ead8e" containerName="istio-proxy" containerID="cri-o://8b99b56c811f2134d1be98443c8a50f3c902b9d17933d66ed91b96e71a8ecb00" gracePeriod=30
Oct 31 03:37:28.533772 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:28.533746    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:37:28.579312 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:28.579287    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:37:29.186606 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:29.186575    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:30.956096 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:30.956059    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:31.169851 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:31.169809    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:31.393457 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:31.393425    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"]
Oct 31 03:37:31.393695 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:31.393671    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh" podUID="186cdfd1-f984-4147-8bea-e13f3e50f892" containerName="istio-proxy" containerID="cri-o://8a27d5002531779c074265e3f5e6886f2b6e885cebac6604c8181922c346e23a" gracePeriod=30
Oct 31 03:37:31.397816 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:31.397759    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"]
Oct 31 03:37:31.398017 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:31.397995    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b" podUID="b0397d82-9c9b-443f-b763-95d71d5adac9" containerName="istio-proxy" containerID="cri-o://6c214fed12307f1e6eef206162a023161df5f51957fba56db9318a570ffc67d7" gracePeriod=30
Oct 31 03:37:31.489499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:31.489417    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:32.254135 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:32.254102    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:32.909424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:32.909388    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:33.284559 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.284507    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:33.295269 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.295241    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:37:33.295398 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.295320    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm" podUID="88fcbedb-2530-4fc4-b9ec-b897333ead8e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:37:33.702226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.702193    4900 generic.go:356] "Generic (PLEG): container finished" podID="88fcbedb-2530-4fc4-b9ec-b897333ead8e" containerID="8b99b56c811f2134d1be98443c8a50f3c902b9d17933d66ed91b96e71a8ecb00" exitCode=0
Oct 31 03:37:33.702417 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.702314    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm" event={"ID":"88fcbedb-2530-4fc4-b9ec-b897333ead8e","Type":"ContainerDied","Data":"8b99b56c811f2134d1be98443c8a50f3c902b9d17933d66ed91b96e71a8ecb00"}
Oct 31 03:37:33.735201 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.735171    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:37:33.857242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.857154    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-envoy\") pod \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") "
Oct 31 03:37:33.857390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.857265    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-data\") pod \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") "
Oct 31 03:37:33.857390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.857299    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-token\" (UniqueName: \"kubernetes.io/projected/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-token\") pod \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") "
Oct 31 03:37:33.857390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.857375    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-workload-socket\") pod \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") "
Oct 31 03:37:33.857525 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.857416    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-podinfo\") pod \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") "
Oct 31 03:37:33.857525 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.857450    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istiod-ca-cert\") pod \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") "
Oct 31 03:37:33.857623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.857538    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88fcbedb-2530-4fc4-b9ec-b897333ead8e-kube-api-access-r4jkh\" (UniqueName: \"kubernetes.io/projected/88fcbedb-2530-4fc4-b9ec-b897333ead8e-kube-api-access-r4jkh\") pod \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") "
Oct 31 03:37:33.857623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.857609    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-credential-socket\") pod \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") "
Oct 31 03:37:33.857733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.857536    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-data" pod "88fcbedb-2530-4fc4-b9ec-b897333ead8e" (UID: "88fcbedb-2530-4fc4-b9ec-b897333ead8e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:33.857733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.857694    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-workload-certs\") pod \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\" (UID: \"88fcbedb-2530-4fc4-b9ec-b897333ead8e\") "
Oct 31 03:37:33.857872 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.857769    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-workload-socket" pod "88fcbedb-2530-4fc4-b9ec-b897333ead8e" (UID: "88fcbedb-2530-4fc4-b9ec-b897333ead8e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:33.858042 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.857988    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-credential-socket" pod "88fcbedb-2530-4fc4-b9ec-b897333ead8e" (UID: "88fcbedb-2530-4fc4-b9ec-b897333ead8e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:33.858042 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.858010    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istiod-ca-cert" pod "88fcbedb-2530-4fc4-b9ec-b897333ead8e" (UID: "88fcbedb-2530-4fc4-b9ec-b897333ead8e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:33.858163 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.858131    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-workload-certs" pod "88fcbedb-2530-4fc4-b9ec-b897333ead8e" (UID: "88fcbedb-2530-4fc4-b9ec-b897333ead8e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:33.858460 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.858442    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.858460 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.858460    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.858589 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.858471    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.858589 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.858484    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.858589 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.858492    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.859642 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.859610    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-podinfo" pod "88fcbedb-2530-4fc4-b9ec-b897333ead8e" (UID: "88fcbedb-2530-4fc4-b9ec-b897333ead8e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:33.859824 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.859776    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88fcbedb-2530-4fc4-b9ec-b897333ead8e-kube-api-access-r4jkh" pod "88fcbedb-2530-4fc4-b9ec-b897333ead8e" (UID: "88fcbedb-2530-4fc4-b9ec-b897333ead8e"). InnerVolumeSpecName "kube-api-access-r4jkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:33.859905 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.859844    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-envoy" pod "88fcbedb-2530-4fc4-b9ec-b897333ead8e" (UID: "88fcbedb-2530-4fc4-b9ec-b897333ead8e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:33.859968 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.859954    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-token" pod "88fcbedb-2530-4fc4-b9ec-b897333ead8e" (UID: "88fcbedb-2530-4fc4-b9ec-b897333ead8e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:33.959418 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.959384    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.959418 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.959412    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.959418 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.959421    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88fcbedb-2530-4fc4-b9ec-b897333ead8e-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:33.959748 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:33.959431    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4jkh\" (UniqueName: \"kubernetes.io/projected/88fcbedb-2530-4fc4-b9ec-b897333ead8e-kube-api-access-r4jkh\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:34.714829 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:34.714043    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm" event={"ID":"88fcbedb-2530-4fc4-b9ec-b897333ead8e","Type":"ContainerDied","Data":"6ce1a3343cbcc3ea6ea15ba209842f259938e82a8de517b9ce9c459f82f4f740"}
Oct 31 03:37:34.715384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:34.714060    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"
Oct 31 03:37:34.715384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:34.714914    4900 scope.go:122] "RemoveContainer" containerID="8b99b56c811f2134d1be98443c8a50f3c902b9d17933d66ed91b96e71a8ecb00"
Oct 31 03:37:34.868520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:34.868486    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"]
Oct 31 03:37:34.914774 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:34.914735    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-lsqtm"]
Oct 31 03:37:35.154278 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:35.154242    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88fcbedb-2530-4fc4-b9ec-b897333ead8e" path="/var/lib/kubelet/pods/88fcbedb-2530-4fc4-b9ec-b897333ead8e/volumes"
Oct 31 03:37:36.629090 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.629054    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:36.698871 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.698844    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:37:36.729799 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.729756    4900 generic.go:356] "Generic (PLEG): container finished" podID="b0397d82-9c9b-443f-b763-95d71d5adac9" containerID="6c214fed12307f1e6eef206162a023161df5f51957fba56db9318a570ffc67d7" exitCode=0
Oct 31 03:37:36.729908 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.729825    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b" event={"ID":"b0397d82-9c9b-443f-b763-95d71d5adac9","Type":"ContainerDied","Data":"6c214fed12307f1e6eef206162a023161df5f51957fba56db9318a570ffc67d7"}
Oct 31 03:37:36.729908 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.729877    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b" event={"ID":"b0397d82-9c9b-443f-b763-95d71d5adac9","Type":"ContainerDied","Data":"7c60da7291ed431af25001caa0b78867a35319813cadb712d04180a65c2dc4c2"}
Oct 31 03:37:36.729908 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.729900    4900 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c60da7291ed431af25001caa0b78867a35319813cadb712d04180a65c2dc4c2"
Oct 31 03:37:36.731304 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.731279    4900 generic.go:356] "Generic (PLEG): container finished" podID="186cdfd1-f984-4147-8bea-e13f3e50f892" containerID="8a27d5002531779c074265e3f5e6886f2b6e885cebac6604c8181922c346e23a" exitCode=0
Oct 31 03:37:36.731396 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.731319    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"
Oct 31 03:37:36.731396 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.731352    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh" event={"ID":"186cdfd1-f984-4147-8bea-e13f3e50f892","Type":"ContainerDied","Data":"8a27d5002531779c074265e3f5e6886f2b6e885cebac6604c8181922c346e23a"}
Oct 31 03:37:36.731479 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.731399    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh" event={"ID":"186cdfd1-f984-4147-8bea-e13f3e50f892","Type":"ContainerDied","Data":"70190f2632af4cea25ac93400406c8189cbca23dfb9be867b769f94475376c25"}
Oct 31 03:37:36.731479 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.731423    4900 scope.go:122] "RemoveContainer" containerID="8a27d5002531779c074265e3f5e6886f2b6e885cebac6604c8181922c346e23a"
Oct 31 03:37:36.735011 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.734994    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:37:36.740355 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.740340    4900 scope.go:122] "RemoveContainer" containerID="8a27d5002531779c074265e3f5e6886f2b6e885cebac6604c8181922c346e23a"
Oct 31 03:37:36.740629 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:37:36.740611    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a27d5002531779c074265e3f5e6886f2b6e885cebac6604c8181922c346e23a\": container with ID starting with 8a27d5002531779c074265e3f5e6886f2b6e885cebac6604c8181922c346e23a not found: ID does not exist" containerID="8a27d5002531779c074265e3f5e6886f2b6e885cebac6604c8181922c346e23a"
Oct 31 03:37:36.740675 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.740639    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a27d5002531779c074265e3f5e6886f2b6e885cebac6604c8181922c346e23a"} err="failed to get container status \"8a27d5002531779c074265e3f5e6886f2b6e885cebac6604c8181922c346e23a\": rpc error: code = NotFound desc = could not find container \"8a27d5002531779c074265e3f5e6886f2b6e885cebac6604c8181922c346e23a\": container with ID starting with 8a27d5002531779c074265e3f5e6886f2b6e885cebac6604c8181922c346e23a not found: ID does not exist"
Oct 31 03:37:36.802317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802291    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-data\") pod \"b0397d82-9c9b-443f-b763-95d71d5adac9\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") "
Oct 31 03:37:36.802465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802336    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-workload-certs\") pod \"186cdfd1-f984-4147-8bea-e13f3e50f892\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") "
Oct 31 03:37:36.802465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802370    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-token\" (UniqueName: \"kubernetes.io/projected/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-token\") pod \"186cdfd1-f984-4147-8bea-e13f3e50f892\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") "
Oct 31 03:37:36.802465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802410    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-data\") pod \"186cdfd1-f984-4147-8bea-e13f3e50f892\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") "
Oct 31 03:37:36.802465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802459    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-podinfo\") pod \"186cdfd1-f984-4147-8bea-e13f3e50f892\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") "
Oct 31 03:37:36.802653 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802498    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-token\" (UniqueName: \"kubernetes.io/projected/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-token\") pod \"b0397d82-9c9b-443f-b763-95d71d5adac9\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") "
Oct 31 03:37:36.802653 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802531    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186cdfd1-f984-4147-8bea-e13f3e50f892-kube-api-access-cpbgd\" (UniqueName: \"kubernetes.io/projected/186cdfd1-f984-4147-8bea-e13f3e50f892-kube-api-access-cpbgd\") pod \"186cdfd1-f984-4147-8bea-e13f3e50f892\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") "
Oct 31 03:37:36.802653 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802567    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-data" pod "b0397d82-9c9b-443f-b763-95d71d5adac9" (UID: "b0397d82-9c9b-443f-b763-95d71d5adac9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.802653 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802572    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-envoy\") pod \"186cdfd1-f984-4147-8bea-e13f3e50f892\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") "
Oct 31 03:37:36.802927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802668    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-workload-certs" pod "186cdfd1-f984-4147-8bea-e13f3e50f892" (UID: "186cdfd1-f984-4147-8bea-e13f3e50f892"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.802927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802706    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-workload-socket\") pod \"186cdfd1-f984-4147-8bea-e13f3e50f892\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") "
Oct 31 03:37:36.802927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802775    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-workload-certs\") pod \"b0397d82-9c9b-443f-b763-95d71d5adac9\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") "
Oct 31 03:37:36.802927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802829    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-workload-socket\") pod \"b0397d82-9c9b-443f-b763-95d71d5adac9\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") "
Oct 31 03:37:36.802927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802830    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-data" pod "186cdfd1-f984-4147-8bea-e13f3e50f892" (UID: "186cdfd1-f984-4147-8bea-e13f3e50f892"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.802927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802886    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0397d82-9c9b-443f-b763-95d71d5adac9-kube-api-access-ktf2r\" (UniqueName: \"kubernetes.io/projected/b0397d82-9c9b-443f-b763-95d71d5adac9-kube-api-access-ktf2r\") pod \"b0397d82-9c9b-443f-b763-95d71d5adac9\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") "
Oct 31 03:37:36.802927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802924    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-credential-socket\") pod \"b0397d82-9c9b-443f-b763-95d71d5adac9\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") "
Oct 31 03:37:36.803286 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.802982    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186cdfd1-f984-4147-8bea-e13f3e50f892-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186cdfd1-f984-4147-8bea-e13f3e50f892-istiod-ca-cert\") pod \"186cdfd1-f984-4147-8bea-e13f3e50f892\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") "
Oct 31 03:37:36.803286 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.803038    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-envoy\") pod \"b0397d82-9c9b-443f-b763-95d71d5adac9\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") "
Oct 31 03:37:36.803286 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.803055    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-workload-socket" pod "186cdfd1-f984-4147-8bea-e13f3e50f892" (UID: "186cdfd1-f984-4147-8bea-e13f3e50f892"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.803286 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.803115    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-credential-socket\") pod \"186cdfd1-f984-4147-8bea-e13f3e50f892\" (UID: \"186cdfd1-f984-4147-8bea-e13f3e50f892\") "
Oct 31 03:37:36.803286 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.803130    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-workload-socket" pod "b0397d82-9c9b-443f-b763-95d71d5adac9" (UID: "b0397d82-9c9b-443f-b763-95d71d5adac9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.803286 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.803215    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0397d82-9c9b-443f-b763-95d71d5adac9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0397d82-9c9b-443f-b763-95d71d5adac9-istiod-ca-cert\") pod \"b0397d82-9c9b-443f-b763-95d71d5adac9\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") "
Oct 31 03:37:36.803286 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.803248    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-podinfo\") pod \"b0397d82-9c9b-443f-b763-95d71d5adac9\" (UID: \"b0397d82-9c9b-443f-b763-95d71d5adac9\") "
Oct 31 03:37:36.803644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.803282    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-workload-certs" pod "b0397d82-9c9b-443f-b763-95d71d5adac9" (UID: "b0397d82-9c9b-443f-b763-95d71d5adac9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.803644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.803324    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-credential-socket" pod "b0397d82-9c9b-443f-b763-95d71d5adac9" (UID: "b0397d82-9c9b-443f-b763-95d71d5adac9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.803644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.803603    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0397d82-9c9b-443f-b763-95d71d5adac9-istiod-ca-cert" pod "b0397d82-9c9b-443f-b763-95d71d5adac9" (UID: "b0397d82-9c9b-443f-b763-95d71d5adac9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:36.803823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.803664    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-credential-socket" pod "186cdfd1-f984-4147-8bea-e13f3e50f892" (UID: "186cdfd1-f984-4147-8bea-e13f3e50f892"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.803977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.803955    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0397d82-9c9b-443f-b763-95d71d5adac9-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.804047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.803983    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.804047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.803999    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.804047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.804011    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.804047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.804031    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.804047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.804045    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.804308 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.804059    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.804308 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.804073    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.804308 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.804087    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.804467 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.804370    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186cdfd1-f984-4147-8bea-e13f3e50f892-istiod-ca-cert" pod "186cdfd1-f984-4147-8bea-e13f3e50f892" (UID: "186cdfd1-f984-4147-8bea-e13f3e50f892"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:36.805387 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.805359    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186cdfd1-f984-4147-8bea-e13f3e50f892-kube-api-access-cpbgd" pod "186cdfd1-f984-4147-8bea-e13f3e50f892" (UID: "186cdfd1-f984-4147-8bea-e13f3e50f892"). InnerVolumeSpecName "kube-api-access-cpbgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:36.805596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.805523    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-token" pod "186cdfd1-f984-4147-8bea-e13f3e50f892" (UID: "186cdfd1-f984-4147-8bea-e13f3e50f892"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:36.805709 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.805664    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-token" pod "b0397d82-9c9b-443f-b763-95d71d5adac9" (UID: "b0397d82-9c9b-443f-b763-95d71d5adac9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:36.805841 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.805810    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0397d82-9c9b-443f-b763-95d71d5adac9-kube-api-access-ktf2r" pod "b0397d82-9c9b-443f-b763-95d71d5adac9" (UID: "b0397d82-9c9b-443f-b763-95d71d5adac9"). InnerVolumeSpecName "kube-api-access-ktf2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:36.805909 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.805880    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-envoy" pod "186cdfd1-f984-4147-8bea-e13f3e50f892" (UID: "186cdfd1-f984-4147-8bea-e13f3e50f892"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.805973 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.805938    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-podinfo" pod "b0397d82-9c9b-443f-b763-95d71d5adac9" (UID: "b0397d82-9c9b-443f-b763-95d71d5adac9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:36.806402 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.806387    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-envoy" pod "b0397d82-9c9b-443f-b763-95d71d5adac9" (UID: "b0397d82-9c9b-443f-b763-95d71d5adac9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:36.806463 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.806425    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-podinfo" pod "186cdfd1-f984-4147-8bea-e13f3e50f892" (UID: "186cdfd1-f984-4147-8bea-e13f3e50f892"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:36.905275 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.905222    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktf2r\" (UniqueName: \"kubernetes.io/projected/b0397d82-9c9b-443f-b763-95d71d5adac9-kube-api-access-ktf2r\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.905275 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.905263    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/186cdfd1-f984-4147-8bea-e13f3e50f892-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.905275 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.905274    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.905275 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.905283    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.905640 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.905292    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.905640 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.905301    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.905640 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.905309    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0397d82-9c9b-443f-b763-95d71d5adac9-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.905640 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.905317    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpbgd\" (UniqueName: \"kubernetes.io/projected/186cdfd1-f984-4147-8bea-e13f3e50f892-kube-api-access-cpbgd\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.905640 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:36.905326    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/186cdfd1-f984-4147-8bea-e13f3e50f892-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.992466 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:37:36.990995    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:37:37.157727 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:37.157690    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"]
Oct 31 03:37:37.157727 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:37.157736    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-nkdvh"]
Oct 31 03:37:37.738102 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:37.738077    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"
Oct 31 03:37:37.938256 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:37.938225    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"]
Oct 31 03:37:37.984064 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:37.984030    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6b3a29ec-openshift-default-5c9596d785-lf64b"]
Oct 31 03:37:38.177205 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:38.177111    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:38.418626 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:38.418597    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:39.153333 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:39.153248    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186cdfd1-f984-4147-8bea-e13f3e50f892" path="/var/lib/kubelet/pods/186cdfd1-f984-4147-8bea-e13f3e50f892/volumes"
Oct 31 03:37:39.153808 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:39.153713    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0397d82-9c9b-443f-b763-95d71d5adac9" path="/var/lib/kubelet/pods/b0397d82-9c9b-443f-b763-95d71d5adac9/volumes"
Oct 31 03:37:39.688631 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:39.688589    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:39.911259 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:39.911227    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:41.486527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.486490    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:41.689908 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.689166    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:41.983344 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.983305    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"]
Oct 31 03:37:41.984161 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984137    4900 cpu_manager.go:414] "containerMap: removing container" podUID="676e6a62-14d6-42a5-81d7-5b08d33de624" containerName="istio-proxy"
Oct 31 03:37:41.984161 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984158    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676e6a62-14d6-42a5-81d7-5b08d33de624" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984172    4900 cpu_manager.go:414] "containerMap: removing container" podUID="186cdfd1-f984-4147-8bea-e13f3e50f892" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984177    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186cdfd1-f984-4147-8bea-e13f3e50f892" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984186    4900 cpu_manager.go:414] "containerMap: removing container" podUID="676e6a62-14d6-42a5-81d7-5b08d33de624" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984191    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676e6a62-14d6-42a5-81d7-5b08d33de624" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984199    4900 cpu_manager.go:414] "containerMap: removing container" podUID="b0397d82-9c9b-443f-b763-95d71d5adac9" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984203    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0397d82-9c9b-443f-b763-95d71d5adac9" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984213    4900 cpu_manager.go:414] "containerMap: removing container" podUID="3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984219    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984236    4900 cpu_manager.go:414] "containerMap: removing container" podUID="88fcbedb-2530-4fc4-b9ec-b897333ead8e" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984240    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88fcbedb-2530-4fc4-b9ec-b897333ead8e" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984325    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="676e6a62-14d6-42a5-81d7-5b08d33de624" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984333    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0397d82-9c9b-443f-b763-95d71d5adac9" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984341    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="3beebd5c-ad9f-4677-ac7c-e5a6ff4c6645" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984353    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="88fcbedb-2530-4fc4-b9ec-b897333ead8e" containerName="istio-proxy"
Oct 31 03:37:41.984360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.984359    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="186cdfd1-f984-4147-8bea-e13f3e50f892" containerName="istio-proxy"
Oct 31 03:37:41.993142 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:41.993107    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"
Oct 31 03:37:42.027844 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.026676    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"]
Oct 31 03:37:42.042448 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.042412    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-329d79\""
Oct 31 03:37:42.068428 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.068389    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-329d79-dockercfg-fjzkj\""
Oct 31 03:37:42.082169 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.082133    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777d67a4-f930-450a-acac-b1b933ceada0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp\" (UID: \"777d67a4-f930-450a-acac-b1b933ceada0\") " pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"
Oct 31 03:37:42.082403 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.082381    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpktw\" (UniqueName: \"kubernetes.io/projected/777d67a4-f930-450a-acac-b1b933ceada0-kube-api-access-gpktw\") pod \"payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp\" (UID: \"777d67a4-f930-450a-acac-b1b933ceada0\") " pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"
Oct 31 03:37:42.112104 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.112074    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:42.183355 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.183321    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpktw\" (UniqueName: \"kubernetes.io/projected/777d67a4-f930-450a-acac-b1b933ceada0-kube-api-access-gpktw\") pod \"payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp\" (UID: \"777d67a4-f930-450a-acac-b1b933ceada0\") " pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"
Oct 31 03:37:42.183541 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.183442    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777d67a4-f930-450a-acac-b1b933ceada0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp\" (UID: \"777d67a4-f930-450a-acac-b1b933ceada0\") " pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"
Oct 31 03:37:42.184145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.184126    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777d67a4-f930-450a-acac-b1b933ceada0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp\" (UID: \"777d67a4-f930-450a-acac-b1b933ceada0\") " pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"
Oct 31 03:37:42.236684 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.236599    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpktw\" (UniqueName: \"kubernetes.io/projected/777d67a4-f930-450a-acac-b1b933ceada0-kube-api-access-gpktw\") pod \"payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp\" (UID: \"777d67a4-f930-450a-acac-b1b933ceada0\") " pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"
Oct 31 03:37:42.321488 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.321459    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"
Oct 31 03:37:42.690489 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.690458    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"]
Oct 31 03:37:42.691190 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:37:42.691159    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777d67a4_f930_450a_acac_b1b933ceada0.slice/crio-ea0d39e3b84bad5fb3ff46a8cae03e3f03a42b264c84faec87a5234b9bd414ff WatchSource:0}: Error finding container ea0d39e3b84bad5fb3ff46a8cae03e3f03a42b264c84faec87a5234b9bd414ff: Status 404 returned error can't find the container with id ea0d39e3b84bad5fb3ff46a8cae03e3f03a42b264c84faec87a5234b9bd414ff
Oct 31 03:37:42.767613 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.767586    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp" event={"ID":"777d67a4-f930-450a-acac-b1b933ceada0","Type":"ContainerStarted","Data":"0b6ec974accf4ce3e4a27982b25c830386ae0d4b97075e20d012c2c0c8698541"}
Oct 31 03:37:42.767733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.767641    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp" event={"ID":"777d67a4-f930-450a-acac-b1b933ceada0","Type":"ContainerStarted","Data":"ea0d39e3b84bad5fb3ff46a8cae03e3f03a42b264c84faec87a5234b9bd414ff"}
Oct 31 03:37:42.767808 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.767767    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"
Oct 31 03:37:42.874338 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:42.874288    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp" podStartSLOduration=1.874274095 podStartE2EDuration="1.874274095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:42.873024082 +0000 UTC m=+1815.839357425" watchObservedRunningTime="2026-10-31 03:37:42.874274095 +0000 UTC m=+1815.840607439"
Oct 31 03:37:45.285045 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:45.285009    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:45.494355 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:45.494320    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:47.427337 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:47.427307    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:47.641728 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:47.641688    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:47.856747 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:47.856701    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:52.391071 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:52.390988    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:53.085020 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:53.084982    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:53.306191 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:53.306158    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:37:53.776282 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:53.776232    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"
Oct 31 03:37:58.370053 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:37:58.370014    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:03.339368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:03.339300    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:06.984029 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:38:06.983990    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:38:08.321224 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:08.321191    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:13.482006 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:13.481967    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:13.723175 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:13.723140    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"]
Oct 31 03:38:13.723447 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:13.723421    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9" podUID="451148a6-ec1b-4620-b650-76f3a578d829" containerName="istio-proxy" containerID="cri-o://47511370eba5edd3a27ebf7004b96c6d1b49607ad9675fcd03dc9e3cec3984d6" gracePeriod=30
Oct 31 03:38:13.769720 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:13.769618    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"]
Oct 31 03:38:13.769940 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:13.769893    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2" podUID="37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" containerName="istio-proxy" containerID="cri-o://8ec86705daab1b45faf1c0128c345f63dd7de1cd9ff6c9455d8bcfda77118cbf" gracePeriod=30
Oct 31 03:38:16.623659 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:16.623625    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:38:16.624148 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:16.623686    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2" podUID="37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:38:16.992614 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:16.992579    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:38:16.992811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:16.992642    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9" podUID="451148a6-ec1b-4620-b650-76f3a578d829" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:38:17.352028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:17.351875    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:17.945009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:17.944971    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:18.976964 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:18.976931    4900 generic.go:356] "Generic (PLEG): container finished" podID="37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" containerID="8ec86705daab1b45faf1c0128c345f63dd7de1cd9ff6c9455d8bcfda77118cbf" exitCode=0
Oct 31 03:38:18.977358 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:18.977119    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2" event={"ID":"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc","Type":"ContainerDied","Data":"8ec86705daab1b45faf1c0128c345f63dd7de1cd9ff6c9455d8bcfda77118cbf"}
Oct 31 03:38:18.978766 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:18.978740    4900 generic.go:356] "Generic (PLEG): container finished" podID="451148a6-ec1b-4620-b650-76f3a578d829" containerID="47511370eba5edd3a27ebf7004b96c6d1b49607ad9675fcd03dc9e3cec3984d6" exitCode=0
Oct 31 03:38:18.978885 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:18.978838    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9" event={"ID":"451148a6-ec1b-4620-b650-76f3a578d829","Type":"ContainerDied","Data":"47511370eba5edd3a27ebf7004b96c6d1b49607ad9675fcd03dc9e3cec3984d6"}
Oct 31 03:38:19.050995 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.050964    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:38:19.081111 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.081089    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:38:19.172620 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.172585    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-workload-certs\") pod \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") "
Oct 31 03:38:19.172817 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.172636    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-credential-socket\") pod \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") "
Oct 31 03:38:19.172817 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.172708    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/451148a6-ec1b-4620-b650-76f3a578d829-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451148a6-ec1b-4620-b650-76f3a578d829-istio-podinfo\") pod \"451148a6-ec1b-4620-b650-76f3a578d829\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") "
Oct 31 03:38:19.172817 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.172728    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-kube-api-access-gt48q\" (UniqueName: \"kubernetes.io/projected/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-kube-api-access-gt48q\") pod \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") "
Oct 31 03:38:19.172993 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.172834    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-istio-envoy\") pod \"451148a6-ec1b-4620-b650-76f3a578d829\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") "
Oct 31 03:38:19.172993 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.172910    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-workload-certs" pod "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" (UID: "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.172993 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.172970    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-credential-socket\") pod \"451148a6-ec1b-4620-b650-76f3a578d829\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") "
Oct 31 03:38:19.173143 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.172998    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-credential-socket" pod "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" (UID: "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.173143 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173042    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/451148a6-ec1b-4620-b650-76f3a578d829-istio-token\" (UniqueName: \"kubernetes.io/projected/451148a6-ec1b-4620-b650-76f3a578d829-istio-token\") pod \"451148a6-ec1b-4620-b650-76f3a578d829\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") "
Oct 31 03:38:19.173143 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173073    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-workload-socket\") pod \"451148a6-ec1b-4620-b650-76f3a578d829\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") "
Oct 31 03:38:19.173143 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173115    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-workload-socket\") pod \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") "
Oct 31 03:38:19.173327 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173169    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istiod-ca-cert\") pod \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") "
Oct 31 03:38:19.173327 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173204    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-credential-socket" pod "451148a6-ec1b-4620-b650-76f3a578d829" (UID: "451148a6-ec1b-4620-b650-76f3a578d829"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.173327 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173225    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/451148a6-ec1b-4620-b650-76f3a578d829-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451148a6-ec1b-4620-b650-76f3a578d829-istiod-ca-cert\") pod \"451148a6-ec1b-4620-b650-76f3a578d829\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") "
Oct 31 03:38:19.173327 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173265    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-podinfo\") pod \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") "
Oct 31 03:38:19.173327 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173306    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-workload-certs\") pod \"451148a6-ec1b-4620-b650-76f3a578d829\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") "
Oct 31 03:38:19.173591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173384    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-istio-data\") pod \"451148a6-ec1b-4620-b650-76f3a578d829\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") "
Oct 31 03:38:19.173591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173412    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-token\" (UniqueName: \"kubernetes.io/projected/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-token\") pod \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") "
Oct 31 03:38:19.173591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173482    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-envoy\") pod \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") "
Oct 31 03:38:19.173591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173515    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-data\") pod \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\" (UID: \"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc\") "
Oct 31 03:38:19.173591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173561    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/451148a6-ec1b-4620-b650-76f3a578d829-kube-api-access-qvn82\" (UniqueName: \"kubernetes.io/projected/451148a6-ec1b-4620-b650-76f3a578d829-kube-api-access-qvn82\") pod \"451148a6-ec1b-4620-b650-76f3a578d829\" (UID: \"451148a6-ec1b-4620-b650-76f3a578d829\") "
Oct 31 03:38:19.173941 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173675    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-workload-socket" pod "451148a6-ec1b-4620-b650-76f3a578d829" (UID: "451148a6-ec1b-4620-b650-76f3a578d829"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.173941 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.173866    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-istio-data" pod "451148a6-ec1b-4620-b650-76f3a578d829" (UID: "451148a6-ec1b-4620-b650-76f3a578d829"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.174262 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.174239    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istiod-ca-cert" pod "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" (UID: "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:38:19.174384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.174292    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.174653 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.174593    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-workload-socket" pod "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" (UID: "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.174653 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.174598    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.174897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.174637    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.174897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.174766    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.174897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.174612    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451148a6-ec1b-4620-b650-76f3a578d829-istiod-ca-cert" pod "451148a6-ec1b-4620-b650-76f3a578d829" (UID: "451148a6-ec1b-4620-b650-76f3a578d829"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:38:19.174897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.174782    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.174897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.174687    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-workload-certs" pod "451148a6-ec1b-4620-b650-76f3a578d829" (UID: "451148a6-ec1b-4620-b650-76f3a578d829"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.174897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.174711    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-data" pod "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" (UID: "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.175407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.175380    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-istio-envoy" pod "451148a6-ec1b-4620-b650-76f3a578d829" (UID: "451148a6-ec1b-4620-b650-76f3a578d829"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.175490 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.175413    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-kube-api-access-gt48q" pod "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" (UID: "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc"). InnerVolumeSpecName "kube-api-access-gt48q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:19.176782 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.176758    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/451148a6-ec1b-4620-b650-76f3a578d829-istio-podinfo" pod "451148a6-ec1b-4620-b650-76f3a578d829" (UID: "451148a6-ec1b-4620-b650-76f3a578d829"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:38:19.176893 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.176804    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451148a6-ec1b-4620-b650-76f3a578d829-istio-token" pod "451148a6-ec1b-4620-b650-76f3a578d829" (UID: "451148a6-ec1b-4620-b650-76f3a578d829"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:19.176893 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.176847    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451148a6-ec1b-4620-b650-76f3a578d829-kube-api-access-qvn82" pod "451148a6-ec1b-4620-b650-76f3a578d829" (UID: "451148a6-ec1b-4620-b650-76f3a578d829"). InnerVolumeSpecName "kube-api-access-qvn82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:19.176893 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.176855    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-podinfo" pod "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" (UID: "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:38:19.177051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.176981    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-token" pod "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" (UID: "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:19.177089 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.177078    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-envoy" pod "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" (UID: "37fe5ead-d281-497c-ab7e-f3fd1d75e5bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.276527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.276432    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/451148a6-ec1b-4620-b650-76f3a578d829-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.276527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.276460    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.276527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.276474    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.276527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.276486    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/451148a6-ec1b-4620-b650-76f3a578d829-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.276527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.276498    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.276527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.276510    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.276527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.276522    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.276918 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.276534    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.276918 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.276547    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.276918 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.276560    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvn82\" (UniqueName: \"kubernetes.io/projected/451148a6-ec1b-4620-b650-76f3a578d829-kube-api-access-qvn82\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.276918 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.276571    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/451148a6-ec1b-4620-b650-76f3a578d829-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.276918 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.276585    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt48q\" (UniqueName: \"kubernetes.io/projected/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc-kube-api-access-gt48q\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.276918 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.276597    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/451148a6-ec1b-4620-b650-76f3a578d829-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.631674 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.631594    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"]
Oct 31 03:38:19.632156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.632142    4900 cpu_manager.go:414] "containerMap: removing container" podUID="37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" containerName="istio-proxy"
Oct 31 03:38:19.632212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.632157    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" containerName="istio-proxy"
Oct 31 03:38:19.632212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.632177    4900 cpu_manager.go:414] "containerMap: removing container" podUID="451148a6-ec1b-4620-b650-76f3a578d829" containerName="istio-proxy"
Oct 31 03:38:19.632212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.632183    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="451148a6-ec1b-4620-b650-76f3a578d829" containerName="istio-proxy"
Oct 31 03:38:19.632310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.632268    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" containerName="istio-proxy"
Oct 31 03:38:19.632310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.632280    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="451148a6-ec1b-4620-b650-76f3a578d829" containerName="istio-proxy"
Oct 31 03:38:19.632310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.632289    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="676e6a62-14d6-42a5-81d7-5b08d33de624" containerName="istio-proxy"
Oct 31 03:38:19.636557 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.636540    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"
Oct 31 03:38:19.682196 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.682159    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b41ee43-543f-4e36-8b24-ca770073064c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4\" (UID: \"4b41ee43-543f-4e36-8b24-ca770073064c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"
Oct 31 03:38:19.682349 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.682317    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk9td\" (UniqueName: \"kubernetes.io/projected/4b41ee43-543f-4e36-8b24-ca770073064c-kube-api-access-gk9td\") pod \"payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4\" (UID: \"4b41ee43-543f-4e36-8b24-ca770073064c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"
Oct 31 03:38:19.685598 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.685573    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8b4030\""
Oct 31 03:38:19.685771 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.685696    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8b4030-dockercfg-4sg5p\""
Oct 31 03:38:19.689266 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.689242    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"]
Oct 31 03:38:19.783999 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.783969    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b41ee43-543f-4e36-8b24-ca770073064c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4\" (UID: \"4b41ee43-543f-4e36-8b24-ca770073064c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"
Oct 31 03:38:19.784173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.784079    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk9td\" (UniqueName: \"kubernetes.io/projected/4b41ee43-543f-4e36-8b24-ca770073064c-kube-api-access-gk9td\") pod \"payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4\" (UID: \"4b41ee43-543f-4e36-8b24-ca770073064c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"
Oct 31 03:38:19.784931 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.784907    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b41ee43-543f-4e36-8b24-ca770073064c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4\" (UID: \"4b41ee43-543f-4e36-8b24-ca770073064c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"
Oct 31 03:38:19.837475 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.837436    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk9td\" (UniqueName: \"kubernetes.io/projected/4b41ee43-543f-4e36-8b24-ca770073064c-kube-api-access-gk9td\") pod \"payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4\" (UID: \"4b41ee43-543f-4e36-8b24-ca770073064c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"
Oct 31 03:38:19.949287 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.949196    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"
Oct 31 03:38:19.986408 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.986363    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"
Oct 31 03:38:19.986408 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.986382    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9" event={"ID":"451148a6-ec1b-4620-b650-76f3a578d829","Type":"ContainerDied","Data":"301832fc2fa441c4f10c068456dcc6908df4c9051fbbecf41512f2d39c08bc88"}
Oct 31 03:38:19.986950 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.986464    4900 scope.go:122] "RemoveContainer" containerID="47511370eba5edd3a27ebf7004b96c6d1b49607ad9675fcd03dc9e3cec3984d6"
Oct 31 03:38:19.987842 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.987821    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"
Oct 31 03:38:19.987940 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.987837    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2" event={"ID":"37fe5ead-d281-497c-ab7e-f3fd1d75e5bc","Type":"ContainerDied","Data":"0e9c4f7b10e973f39455601fe9c4e5120fc05a41aa636c1e4e50a342b68728db"}
Oct 31 03:38:19.997669 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:19.997643    4900 scope.go:122] "RemoveContainer" containerID="8ec86705daab1b45faf1c0128c345f63dd7de1cd9ff6c9455d8bcfda77118cbf"
Oct 31 03:38:20.142144 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:20.142112    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"]
Oct 31 03:38:20.146860 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:20.146834    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-j8gl2"]
Oct 31 03:38:20.250035 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:20.250001    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"]
Oct 31 03:38:20.254288 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:20.254257    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qz8k9"]
Oct 31 03:38:20.354382 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:20.354354    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"]
Oct 31 03:38:20.356102 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:38:20.356078    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b41ee43_543f_4e36_8b24_ca770073064c.slice/crio-9cde85f69b5a70d74b8603ac3f3c24a39ad28a5fd6a39c86a663db98896bf7ef WatchSource:0}: Error finding container 9cde85f69b5a70d74b8603ac3f3c24a39ad28a5fd6a39c86a663db98896bf7ef: Status 404 returned error can't find the container with id 9cde85f69b5a70d74b8603ac3f3c24a39ad28a5fd6a39c86a663db98896bf7ef
Oct 31 03:38:20.358461 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:20.358444    4900 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:38:20.800282 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:20.800228    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:21.002248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:21.002213    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4" event={"ID":"4b41ee43-543f-4e36-8b24-ca770073064c","Type":"ContainerStarted","Data":"9cde85f69b5a70d74b8603ac3f3c24a39ad28a5fd6a39c86a663db98896bf7ef"}
Oct 31 03:38:21.154238 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:21.154147    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37fe5ead-d281-497c-ab7e-f3fd1d75e5bc" path="/var/lib/kubelet/pods/37fe5ead-d281-497c-ab7e-f3fd1d75e5bc/volumes"
Oct 31 03:38:21.154826 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:21.154808    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="451148a6-ec1b-4620-b650-76f3a578d829" path="/var/lib/kubelet/pods/451148a6-ec1b-4620-b650-76f3a578d829/volumes"
Oct 31 03:38:23.019301 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:23.019266    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4" event={"ID":"4b41ee43-543f-4e36-8b24-ca770073064c","Type":"ContainerStarted","Data":"323fc83703c39f9efc966f9bbdfd050b6763f53ab775bd7c027a3850dbd8ec41"}
Oct 31 03:38:23.019726 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:23.019385    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"
Oct 31 03:38:23.128593 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:23.128529    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4" podStartSLOduration=1.9820380370000001 podStartE2EDuration="4.128515526s" totalImagesPullingTime="2.146477489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:23.127393793 +0000 UTC m=+1856.093727138" watchObservedRunningTime="2026-10-31 03:38:23.128515526 +0000 UTC m=+1856.094848866"
Oct 31 03:38:24.226634 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:24.226596    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"]
Oct 31 03:38:24.227078 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:24.226840    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerName="istio-proxy" containerID="cri-o://691e1d32cae276ddfeb59a09a47dd3542a8d48755fd6d8c60e1e10e225f8a55b" gracePeriod=30
Oct 31 03:38:25.178879 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:25.178849    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:25.843443 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:25.843352    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:29.492675 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.492607    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:38:29.616182 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.616140    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-workload-socket\") pod \"2b1b3d8d-e144-44ed-8efa-193251bd599c\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") "
Oct 31 03:38:29.616375 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.616199    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b1b3d8d-e144-44ed-8efa-193251bd599c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1b3d8d-e144-44ed-8efa-193251bd599c-istiod-ca-cert\") pod \"2b1b3d8d-e144-44ed-8efa-193251bd599c\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") "
Oct 31 03:38:29.616375 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.616263    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-token\" (UniqueName: \"kubernetes.io/projected/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-token\") pod \"2b1b3d8d-e144-44ed-8efa-193251bd599c\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") "
Oct 31 03:38:29.616494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.616414    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-credential-socket\") pod \"2b1b3d8d-e144-44ed-8efa-193251bd599c\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") "
Oct 31 03:38:29.616557 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.616498    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1b3d8d-e144-44ed-8efa-193251bd599c-kube-api-access-wj8nv\" (UniqueName: \"kubernetes.io/projected/2b1b3d8d-e144-44ed-8efa-193251bd599c-kube-api-access-wj8nv\") pod \"2b1b3d8d-e144-44ed-8efa-193251bd599c\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") "
Oct 31 03:38:29.616557 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.616421    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-workload-socket" pod "2b1b3d8d-e144-44ed-8efa-193251bd599c" (UID: "2b1b3d8d-e144-44ed-8efa-193251bd599c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:29.616658 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.616587    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1b3d8d-e144-44ed-8efa-193251bd599c-istiod-ca-cert" pod "2b1b3d8d-e144-44ed-8efa-193251bd599c" (UID: "2b1b3d8d-e144-44ed-8efa-193251bd599c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:38:29.616658 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.616597    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-credential-socket" pod "2b1b3d8d-e144-44ed-8efa-193251bd599c" (UID: "2b1b3d8d-e144-44ed-8efa-193251bd599c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:29.616658 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.616604    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-podinfo\") pod \"2b1b3d8d-e144-44ed-8efa-193251bd599c\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") "
Oct 31 03:38:29.616905 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.616677    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-data\") pod \"2b1b3d8d-e144-44ed-8efa-193251bd599c\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") "
Oct 31 03:38:29.616905 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.616775    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-workload-certs\") pod \"2b1b3d8d-e144-44ed-8efa-193251bd599c\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") "
Oct 31 03:38:29.616905 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.616835    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-envoy\") pod \"2b1b3d8d-e144-44ed-8efa-193251bd599c\" (UID: \"2b1b3d8d-e144-44ed-8efa-193251bd599c\") "
Oct 31 03:38:29.617070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.616961    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-data" pod "2b1b3d8d-e144-44ed-8efa-193251bd599c" (UID: "2b1b3d8d-e144-44ed-8efa-193251bd599c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:29.617070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.617006    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-workload-certs" pod "2b1b3d8d-e144-44ed-8efa-193251bd599c" (UID: "2b1b3d8d-e144-44ed-8efa-193251bd599c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:29.617395 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.617372    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:29.617485 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.617401    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:29.617485 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.617417    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:29.617485 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.617432    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1b3d8d-e144-44ed-8efa-193251bd599c-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:29.617485 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.617445    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:29.618585 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.618556    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-token" pod "2b1b3d8d-e144-44ed-8efa-193251bd599c" (UID: "2b1b3d8d-e144-44ed-8efa-193251bd599c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:29.618692 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.618672    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-podinfo" pod "2b1b3d8d-e144-44ed-8efa-193251bd599c" (UID: "2b1b3d8d-e144-44ed-8efa-193251bd599c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:38:29.618881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.618861    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1b3d8d-e144-44ed-8efa-193251bd599c-kube-api-access-wj8nv" pod "2b1b3d8d-e144-44ed-8efa-193251bd599c" (UID: "2b1b3d8d-e144-44ed-8efa-193251bd599c"). InnerVolumeSpecName "kube-api-access-wj8nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:29.618881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.618874    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-envoy" pod "2b1b3d8d-e144-44ed-8efa-193251bd599c" (UID: "2b1b3d8d-e144-44ed-8efa-193251bd599c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:29.718745 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.718711    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:29.718745 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.718741    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:29.718745 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.718750    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1b3d8d-e144-44ed-8efa-193251bd599c-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:29.718994 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:29.718759    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj8nv\" (UniqueName: \"kubernetes.io/projected/2b1b3d8d-e144-44ed-8efa-193251bd599c-kube-api-access-wj8nv\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:30.064911 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:30.064875    4900 generic.go:356] "Generic (PLEG): container finished" podID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerID="691e1d32cae276ddfeb59a09a47dd3542a8d48755fd6d8c60e1e10e225f8a55b" exitCode=0
Oct 31 03:38:30.065124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:30.064924    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"
Oct 31 03:38:30.065124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:30.064954    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" event={"ID":"2b1b3d8d-e144-44ed-8efa-193251bd599c","Type":"ContainerDied","Data":"691e1d32cae276ddfeb59a09a47dd3542a8d48755fd6d8c60e1e10e225f8a55b"}
Oct 31 03:38:30.065124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:30.065019    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd" event={"ID":"2b1b3d8d-e144-44ed-8efa-193251bd599c","Type":"ContainerDied","Data":"8980576e52db555edef763bbe785a2d87c5e364c2d2320fe25e5372bbfc0150a"}
Oct 31 03:38:30.065124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:30.065051    4900 scope.go:122] "RemoveContainer" containerID="691e1d32cae276ddfeb59a09a47dd3542a8d48755fd6d8c60e1e10e225f8a55b"
Oct 31 03:38:30.075779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:30.075761    4900 scope.go:122] "RemoveContainer" containerID="691e1d32cae276ddfeb59a09a47dd3542a8d48755fd6d8c60e1e10e225f8a55b"
Oct 31 03:38:30.076061 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:38:30.076040    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691e1d32cae276ddfeb59a09a47dd3542a8d48755fd6d8c60e1e10e225f8a55b\": container with ID starting with 691e1d32cae276ddfeb59a09a47dd3542a8d48755fd6d8c60e1e10e225f8a55b not found: ID does not exist" containerID="691e1d32cae276ddfeb59a09a47dd3542a8d48755fd6d8c60e1e10e225f8a55b"
Oct 31 03:38:30.076140 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:30.076072    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691e1d32cae276ddfeb59a09a47dd3542a8d48755fd6d8c60e1e10e225f8a55b"} err="failed to get container status \"691e1d32cae276ddfeb59a09a47dd3542a8d48755fd6d8c60e1e10e225f8a55b\": rpc error: code = NotFound desc = could not find container \"691e1d32cae276ddfeb59a09a47dd3542a8d48755fd6d8c60e1e10e225f8a55b\": container with ID starting with 691e1d32cae276ddfeb59a09a47dd3542a8d48755fd6d8c60e1e10e225f8a55b not found: ID does not exist"
Oct 31 03:38:30.218837 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:30.218803    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"]
Oct 31 03:38:30.264955 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:30.264924    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-ddsxd"]
Oct 31 03:38:31.154975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:31.154945    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" path="/var/lib/kubelet/pods/2b1b3d8d-e144-44ed-8efa-193251bd599c/volumes"
Oct 31 03:38:32.698125 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:32.698081    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:34.026969 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:34.026934    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"
Oct 31 03:38:36.987009 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:38:36.986970    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:38:37.793576 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:37.793541    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:38.254748 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.254716    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"]
Oct 31 03:38:38.255260 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.255246    4900 cpu_manager.go:414] "containerMap: removing container" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerName="istio-proxy"
Oct 31 03:38:38.255305 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.255261    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerName="istio-proxy"
Oct 31 03:38:38.255371 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.255362    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b1b3d8d-e144-44ed-8efa-193251bd599c" containerName="istio-proxy"
Oct 31 03:38:38.258772 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.258753    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.325024 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.324984    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-data\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.325267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.325041    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-token\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.325267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.325195    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-envoy\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.325267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.325234    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb5n6\" (UniqueName: \"kubernetes.io/projected/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-kube-api-access-mb5n6\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.325506 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.325270    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istiod-ca-cert\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.325506 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.325333    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-credential-socket\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.325506 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.325449    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-workload-certs\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.325506 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.325489    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-podinfo\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.325823 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.325518    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-workload-socket\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.373244 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.373210    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"]
Oct 31 03:38:38.383847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.380346    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c35264-openshift-default-dockercfg-vs2qw\""
Oct 31 03:38:38.426606 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.426572    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-workload-certs\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.426839 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.426615    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-podinfo\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.426839 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.426636    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-workload-socket\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.426839 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.426727    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-data\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.427012 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.426837    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-token\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.427012 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.426915    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-envoy\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.427012 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.426935    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb5n6\" (UniqueName: \"kubernetes.io/projected/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-kube-api-access-mb5n6\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.427012 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.426967    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istiod-ca-cert\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.427202 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.427016    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-credential-socket\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.427299 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.427276    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-data\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.427538 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.427513    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-workload-certs\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.427635 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.427556    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-credential-socket\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.427725 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.427708    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-workload-socket\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.427824 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.427804    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istiod-ca-cert\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.429286 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.429264    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-podinfo\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.429376 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.429294    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-envoy\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.481269 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.481190    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-token\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.482635 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.482606    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb5n6\" (UniqueName: \"kubernetes.io/projected/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-kube-api-access-mb5n6\") pod \"e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") " pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.500623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.500593    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:38.572580 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.572494    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:38.865176 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.865102    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:38.988851 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.988821    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"]
Oct 31 03:38:38.989843 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:38:38.989816    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e56ee62_55fe_4c65_98d2_15ef41f72bb5.slice/crio-e2768800cb948feadc5d9c33e219b5d3ec460b85b125d331efd8cb8134d4f75a WatchSource:0}: Error finding container e2768800cb948feadc5d9c33e219b5d3ec460b85b125d331efd8cb8134d4f75a: Status 404 returned error can't find the container with id e2768800cb948feadc5d9c33e219b5d3ec460b85b125d331efd8cb8134d4f75a
Oct 31 03:38:38.992631 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.992597    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:38.992733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.992662    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:38.992733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:38.992693    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:39.119466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:39.119380    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs" event={"ID":"0e56ee62-55fe-4c65-98d2-15ef41f72bb5","Type":"ContainerStarted","Data":"c1cc9234a64c15ae7fdb69bd6013b6720b0921391d19e5d5b39e085a8446b87f"}
Oct 31 03:38:39.119466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:39.119441    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs" event={"ID":"0e56ee62-55fe-4c65-98d2-15ef41f72bb5","Type":"ContainerStarted","Data":"e2768800cb948feadc5d9c33e219b5d3ec460b85b125d331efd8cb8134d4f75a"}
Oct 31 03:38:39.231338 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:39.231281    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs" podStartSLOduration=1.231267503 podStartE2EDuration="1.231267503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:39.228069537 +0000 UTC m=+1872.194402881" watchObservedRunningTime="2026-10-31 03:38:39.231267503 +0000 UTC m=+1872.197600847"
Oct 31 03:38:39.497310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:39.497275    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:39.573523 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:39.573490    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:40.578445 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:40.578413    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:41.132625 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:41.132590    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:41.133700 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:41.133678    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:38:42.169136 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:42.169096    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:42.384546 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:42.384511    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:44.526780 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.526743    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"]
Oct 31 03:38:44.530466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.530444    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:44.579103 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.579075    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-329d79-serving-cert\""
Oct 31 03:38:44.609359 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.609330    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ff4be5-188e-4c34-9627-fad9f2a7f296-openshift-service-ca\") pod \"maas-api-e2e-api-a-329d79-89d686b99-f9mv9\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:44.609624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.609473    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v98q\" (UniqueName: \"kubernetes.io/projected/89ff4be5-188e-4c34-9627-fad9f2a7f296-kube-api-access-6v98q\") pod \"maas-api-e2e-api-a-329d79-89d686b99-f9mv9\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:44.609624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.609507    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-metrics-certs\") pod \"maas-api-e2e-api-a-329d79-89d686b99-f9mv9\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:44.609624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.609593    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-maas-api-tls\") pod \"maas-api-e2e-api-a-329d79-89d686b99-f9mv9\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:44.628167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.628140    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"]
Oct 31 03:38:44.711009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.710976    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ff4be5-188e-4c34-9627-fad9f2a7f296-openshift-service-ca\") pod \"maas-api-e2e-api-a-329d79-89d686b99-f9mv9\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:44.711212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.711073    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v98q\" (UniqueName: \"kubernetes.io/projected/89ff4be5-188e-4c34-9627-fad9f2a7f296-kube-api-access-6v98q\") pod \"maas-api-e2e-api-a-329d79-89d686b99-f9mv9\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:44.711212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.711097    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-metrics-certs\") pod \"maas-api-e2e-api-a-329d79-89d686b99-f9mv9\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:44.711212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.711119    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-maas-api-tls\") pod \"maas-api-e2e-api-a-329d79-89d686b99-f9mv9\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:44.711376 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:38:44.711324    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-serving-cert: secret "maas-api-e2e-api-a-329d79-serving-cert" not found
Oct 31 03:38:44.711428 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:38:44.711393    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-maas-api-tls podName:89ff4be5-188e-4c34-9627-fad9f2a7f296 nodeName:}" failed. No retries permitted until 2026-10-31 03:38:45.211376826 +0000 UTC m=+1878.177710148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-maas-api-tls") pod "maas-api-e2e-api-a-329d79-89d686b99-f9mv9" (UID: "89ff4be5-188e-4c34-9627-fad9f2a7f296") : secret "maas-api-e2e-api-a-329d79-serving-cert" not found
Oct 31 03:38:44.711712 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.711689    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ff4be5-188e-4c34-9627-fad9f2a7f296-openshift-service-ca\") pod \"maas-api-e2e-api-a-329d79-89d686b99-f9mv9\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:44.713465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.713446    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-metrics-certs\") pod \"maas-api-e2e-api-a-329d79-89d686b99-f9mv9\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:44.767466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.767431    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v98q\" (UniqueName: \"kubernetes.io/projected/89ff4be5-188e-4c34-9627-fad9f2a7f296-kube-api-access-6v98q\") pod \"maas-api-e2e-api-a-329d79-89d686b99-f9mv9\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:44.789443 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.789354    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:44.994098 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:44.994061    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:45.219627 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:45.219535    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-maas-api-tls\") pod \"maas-api-e2e-api-a-329d79-89d686b99-f9mv9\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:45.222175 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:45.222150    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-maas-api-tls\") pod \"maas-api-e2e-api-a-329d79-89d686b99-f9mv9\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:45.444768 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:45.444725    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:45.811995 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:45.811968    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"]
Oct 31 03:38:45.813630 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:38:45.813601    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ff4be5_188e_4c34_9627_fad9f2a7f296.slice/crio-35a1a23fc586e9ae0eccd2194def3c3a3f907b33ade77ad6cb9867e4c89be0b4 WatchSource:0}: Error finding container 35a1a23fc586e9ae0eccd2194def3c3a3f907b33ade77ad6cb9867e4c89be0b4: Status 404 returned error can't find the container with id 35a1a23fc586e9ae0eccd2194def3c3a3f907b33ade77ad6cb9867e4c89be0b4
Oct 31 03:38:46.161950 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:46.161865    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9" event={"ID":"89ff4be5-188e-4c34-9627-fad9f2a7f296","Type":"ContainerStarted","Data":"35a1a23fc586e9ae0eccd2194def3c3a3f907b33ade77ad6cb9867e4c89be0b4"}
Oct 31 03:38:46.452670 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:46.452573    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:47.171687 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:47.171589    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9" event={"ID":"89ff4be5-188e-4c34-9627-fad9f2a7f296","Type":"ContainerStarted","Data":"dd5a50d19b99054927636b54c920f1d4b6259fb955c2cd681061b94ae91168f7"}
Oct 31 03:38:47.171687 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:47.171652    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:47.283010 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:47.282954    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9" podStartSLOduration=2.232920329 podStartE2EDuration="3.282939157s" totalImagesPullingTime="1.050018828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:47.280342799 +0000 UTC m=+1880.246676143" watchObservedRunningTime="2026-10-31 03:38:47.282939157 +0000 UTC m=+1880.249272500"
Oct 31 03:38:48.800690 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:48.800655    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:50.317350 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:50.317317    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:50.954399 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:50.954369    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:51.167801 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.167746    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:51.279834 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.279801    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"]
Oct 31 03:38:51.283950 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.283934    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"
Oct 31 03:38:51.379295 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.379239    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c35264-dockercfg-fgdwp\""
Oct 31 03:38:51.380720 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.380690    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c35264\""
Oct 31 03:38:51.385968 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.385933    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"]
Oct 31 03:38:51.421599 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.421563    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c931b976-b5fd-4c17-8f77-89a726ff17d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw\" (UID: \"c931b976-b5fd-4c17-8f77-89a726ff17d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"
Oct 31 03:38:51.421817 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.421732    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c75z8\" (UniqueName: \"kubernetes.io/projected/c931b976-b5fd-4c17-8f77-89a726ff17d8-kube-api-access-c75z8\") pod \"payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw\" (UID: \"c931b976-b5fd-4c17-8f77-89a726ff17d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"
Oct 31 03:38:51.469693 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.469655    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"]
Oct 31 03:38:51.475135 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.475109    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"
Oct 31 03:38:51.522681 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.522644    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c75z8\" (UniqueName: \"kubernetes.io/projected/c931b976-b5fd-4c17-8f77-89a726ff17d8-kube-api-access-c75z8\") pod \"payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw\" (UID: \"c931b976-b5fd-4c17-8f77-89a726ff17d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"
Oct 31 03:38:51.522896 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.522691    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q5rz\" (UniqueName: \"kubernetes.io/projected/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544-kube-api-access-2q5rz\") pod \"payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m\" (UID: \"4b9ab7b8-5f81-4b67-b7c3-1dee661bb544\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"
Oct 31 03:38:51.522896 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.522742    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m\" (UID: \"4b9ab7b8-5f81-4b67-b7c3-1dee661bb544\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"
Oct 31 03:38:51.522980 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.522940    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c931b976-b5fd-4c17-8f77-89a726ff17d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw\" (UID: \"c931b976-b5fd-4c17-8f77-89a726ff17d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"
Oct 31 03:38:51.523609 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.523589    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c931b976-b5fd-4c17-8f77-89a726ff17d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw\" (UID: \"c931b976-b5fd-4c17-8f77-89a726ff17d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"
Oct 31 03:38:51.531390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.531322    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"]
Oct 31 03:38:51.584047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.584019    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c75z8\" (UniqueName: \"kubernetes.io/projected/c931b976-b5fd-4c17-8f77-89a726ff17d8-kube-api-access-c75z8\") pod \"payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw\" (UID: \"c931b976-b5fd-4c17-8f77-89a726ff17d8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"
Oct 31 03:38:51.597093 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.597064    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"
Oct 31 03:38:51.623937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.623904    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5rz\" (UniqueName: \"kubernetes.io/projected/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544-kube-api-access-2q5rz\") pod \"payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m\" (UID: \"4b9ab7b8-5f81-4b67-b7c3-1dee661bb544\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"
Oct 31 03:38:51.624080 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.623967    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m\" (UID: \"4b9ab7b8-5f81-4b67-b7c3-1dee661bb544\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"
Oct 31 03:38:51.624781 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.624757    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m\" (UID: \"4b9ab7b8-5f81-4b67-b7c3-1dee661bb544\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"
Oct 31 03:38:51.678757 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.678726    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q5rz\" (UniqueName: \"kubernetes.io/projected/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544-kube-api-access-2q5rz\") pod \"payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m\" (UID: \"4b9ab7b8-5f81-4b67-b7c3-1dee661bb544\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"
Oct 31 03:38:51.789991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.789910    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"
Oct 31 03:38:51.966864 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:51.966837    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"]
Oct 31 03:38:51.968304 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:38:51.968279    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc931b976_b5fd_4c17_8f77_89a726ff17d8.slice/crio-c91065c241393b711e6a9e742c4f3e5b4d9a5301da4d8d38a6316f3518ec8062 WatchSource:0}: Error finding container c91065c241393b711e6a9e742c4f3e5b4d9a5301da4d8d38a6316f3518ec8062: Status 404 returned error can't find the container with id c91065c241393b711e6a9e742c4f3e5b4d9a5301da4d8d38a6316f3518ec8062
Oct 31 03:38:52.154726 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:52.154699    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"]
Oct 31 03:38:52.156457 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:38:52.156431    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9ab7b8_5f81_4b67_b7c3_1dee661bb544.slice/crio-87383da9d0da6f728d36a0c1b2b9193e592c31f11aa33168b549203e8c904db2 WatchSource:0}: Error finding container 87383da9d0da6f728d36a0c1b2b9193e592c31f11aa33168b549203e8c904db2: Status 404 returned error can't find the container with id 87383da9d0da6f728d36a0c1b2b9193e592c31f11aa33168b549203e8c904db2
Oct 31 03:38:52.220196 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:52.220161    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw" event={"ID":"c931b976-b5fd-4c17-8f77-89a726ff17d8","Type":"ContainerStarted","Data":"2b11d2330155350540f1b3433c7dc538d0063438f29c91e96ccb58565b7d2bb2"}
Oct 31 03:38:52.220389 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:52.220371    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"
Oct 31 03:38:52.220485 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:52.220411    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw" event={"ID":"c931b976-b5fd-4c17-8f77-89a726ff17d8","Type":"ContainerStarted","Data":"c91065c241393b711e6a9e742c4f3e5b4d9a5301da4d8d38a6316f3518ec8062"}
Oct 31 03:38:52.221637 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:52.221614    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m" event={"ID":"4b9ab7b8-5f81-4b67-b7c3-1dee661bb544","Type":"ContainerStarted","Data":"87383da9d0da6f728d36a0c1b2b9193e592c31f11aa33168b549203e8c904db2"}
Oct 31 03:38:52.328064 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:52.328011    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw" podStartSLOduration=1.327996765 podStartE2EDuration="1.327996765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:52.326324769 +0000 UTC m=+1885.292658114" watchObservedRunningTime="2026-10-31 03:38:52.327996765 +0000 UTC m=+1885.294330131"
Oct 31 03:38:52.415381 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:52.415305    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:52.626969 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:52.626934    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:52.842081 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:52.842043    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:53.191735 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:53.191653    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:38:53.229160 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:53.229126    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m" event={"ID":"4b9ab7b8-5f81-4b67-b7c3-1dee661bb544","Type":"ContainerStarted","Data":"62f8957b47962a285b094fa4e5191ecf5758af9a1c5988d80467c6e82e30255c"}
Oct 31 03:38:53.405572 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:53.405510    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m" podStartSLOduration=2.405496016 podStartE2EDuration="2.405496016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:53.404561271 +0000 UTC m=+1886.370894615" watchObservedRunningTime="2026-10-31 03:38:53.405496016 +0000 UTC m=+1886.371829360"
Oct 31 03:38:54.236088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:54.236057    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"
Oct 31 03:38:54.870480 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:54.870445    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:38:55.868181 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:38:55.868141    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:01.199221 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:01.199181    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:03.233024 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:03.232997    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"
Oct 31 03:39:05.245088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:05.245060    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"
Oct 31 03:39:06.392394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:06.392355    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:06.986825 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:39:06.986772    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:39:08.389123 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:08.389083    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:10.215364 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:10.215282    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:10.697919 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:10.697762    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:10.905923 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:10.905893    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:12.228373 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:12.228307    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:16.270129 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:16.270048    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:16.488940 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:16.488909    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:16.709009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:16.708902    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:17.845980 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:17.845941    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:23.570325 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:23.570278    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:25.942012 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:25.941979    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:26.297421 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:26.297385    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:28.905905 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:28.905852    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:35.489190 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:35.489156    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:36.985008 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:39:36.984972    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:39:39.785297 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:39.785266    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:40.327475 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:40.327442    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:40.547195 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:40.547152    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:41.051394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:41.051343    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:41.595413 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:41.595369    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:41.805956 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:41.805928    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:43.174991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:43.174956    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:45.717926 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:45.717887    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:45.936640 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:45.936607    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:46.852414 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:46.852382    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:47.288131 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:47.288088    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:47.507577 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:47.507545    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:47.811897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:47.811865    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:48.449008 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:48.448980    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:48.666069 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:48.666034    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:50.079841 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:50.079799    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:50.431273 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:50.431178    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:50.499722 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:50.499691    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"]
Oct 31 03:39:50.499938 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:50.499918    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr" podUID="7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" containerName="istio-proxy" containerID="cri-o://9a3bd92b9ab6c5a37bf09bf98c30825cc0a688802a3c3882fcd77992aa71f175" gracePeriod=30
Oct 31 03:39:50.650754 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:50.648546    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:51.397759 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:51.397723    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:52.549486 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:52.549448    4900 patch_prober.go:29] interesting pod/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:39:52.550007 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:52.549520    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr" podUID="7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:39:53.062181 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:53.062142    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:53.314331 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:53.314241    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:54.935305 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:54.935268    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:55.763751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.763679    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:39:55.811678 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.811640    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-envoy\") pod \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") "
Oct 31 03:39:55.811883 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.811704    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-kube-api-access-qvvfm\" (UniqueName: \"kubernetes.io/projected/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-kube-api-access-qvvfm\") pod \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") "
Oct 31 03:39:55.811883 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.811783    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istiod-ca-cert\") pod \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") "
Oct 31 03:39:55.811883 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.811870    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-data\") pod \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") "
Oct 31 03:39:55.812074 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.811928    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-podinfo\") pod \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") "
Oct 31 03:39:55.812074 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.811953    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-token\" (UniqueName: \"kubernetes.io/projected/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-token\") pod \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") "
Oct 31 03:39:55.812074 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.811983    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-workload-certs\") pod \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") "
Oct 31 03:39:55.812238 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.812091    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-workload-socket\") pod \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") "
Oct 31 03:39:55.812238 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.812134    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-credential-socket\") pod \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\" (UID: \"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5\") "
Oct 31 03:39:55.812361 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.812235    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-data" pod "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" (UID: "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.812361 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.812312    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istiod-ca-cert" pod "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" (UID: "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:55.812472 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.812419    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-workload-certs" pod "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" (UID: "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.812472 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.812454    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-workload-socket" pod "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" (UID: "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.812994 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.812618    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-credential-socket" pod "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" (UID: "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.812994 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.812895    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.812994 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.812917    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.812994 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.812933    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.812994 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.812949    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.812994 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.812965    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.814159 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.814135    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-podinfo" pod "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" (UID: "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:39:55.814381 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.814225    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-envoy" pod "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" (UID: "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:55.815198 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.815174    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-kube-api-access-qvvfm" pod "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" (UID: "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5"). InnerVolumeSpecName "kube-api-access-qvvfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:55.815294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.815264    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-token" pod "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" (UID: "7745c0a4-53b5-40f9-9b78-b4d510f0c6c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:55.832829 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.831627    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:39:55.914507 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.914471    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.914507 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.914504    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvvfm\" (UniqueName: \"kubernetes.io/projected/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-kube-api-access-qvvfm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.914507 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.914515    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:55.914751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:55.914524    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:56.619764 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:56.619728    4900 generic.go:356] "Generic (PLEG): container finished" podID="7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" containerID="9a3bd92b9ab6c5a37bf09bf98c30825cc0a688802a3c3882fcd77992aa71f175" exitCode=0
Oct 31 03:39:56.620265 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:56.619808    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr" event={"ID":"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5","Type":"ContainerDied","Data":"9a3bd92b9ab6c5a37bf09bf98c30825cc0a688802a3c3882fcd77992aa71f175"}
Oct 31 03:39:56.620265 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:56.619864    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr" event={"ID":"7745c0a4-53b5-40f9-9b78-b4d510f0c6c5","Type":"ContainerDied","Data":"c0db1ffdd8c2e6b362175c12a54d240db29217780674a9e3b613bb3e2d1bd610"}
Oct 31 03:39:56.620265 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:56.619887    4900 scope.go:122] "RemoveContainer" containerID="9a3bd92b9ab6c5a37bf09bf98c30825cc0a688802a3c3882fcd77992aa71f175"
Oct 31 03:39:56.620265 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:56.619814    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"
Oct 31 03:39:56.630048 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:56.630028    4900 scope.go:122] "RemoveContainer" containerID="9a3bd92b9ab6c5a37bf09bf98c30825cc0a688802a3c3882fcd77992aa71f175"
Oct 31 03:39:56.630306 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:39:56.630282    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a3bd92b9ab6c5a37bf09bf98c30825cc0a688802a3c3882fcd77992aa71f175\": container with ID starting with 9a3bd92b9ab6c5a37bf09bf98c30825cc0a688802a3c3882fcd77992aa71f175 not found: ID does not exist" containerID="9a3bd92b9ab6c5a37bf09bf98c30825cc0a688802a3c3882fcd77992aa71f175"
Oct 31 03:39:56.630374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:56.630315    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a3bd92b9ab6c5a37bf09bf98c30825cc0a688802a3c3882fcd77992aa71f175"} err="failed to get container status \"9a3bd92b9ab6c5a37bf09bf98c30825cc0a688802a3c3882fcd77992aa71f175\": rpc error: code = NotFound desc = could not find container \"9a3bd92b9ab6c5a37bf09bf98c30825cc0a688802a3c3882fcd77992aa71f175\": container with ID starting with 9a3bd92b9ab6c5a37bf09bf98c30825cc0a688802a3c3882fcd77992aa71f175 not found: ID does not exist"
Oct 31 03:39:56.820914 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:56.820879    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"]
Oct 31 03:39:56.867172 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:56.867139    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1aad0-openshift-default-9dcd96699-sl8fr"]
Oct 31 03:39:57.153496 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:57.153464    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" path="/var/lib/kubelet/pods/7745c0a4-53b5-40f9-9b78-b4d510f0c6c5/volumes"
Oct 31 03:39:57.242422 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:39:57.242392    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:01.397771 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:01.397741    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:04.072704 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:04.072668    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:04.275229 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:04.275199    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:05.874010 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:05.873980    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:06.734011 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:06.733967    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:06.938438 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:06.938407    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:06.985231 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:40:06.985147    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:40:07.389301 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:07.389214    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:07.616442 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:07.616389    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:11.392393 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:11.392356    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:11.692966 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:11.692883    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:11.927819 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:11.925830    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:13.734055 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:13.734010    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:14.987083 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:14.987044    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:15.194910 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:15.194867    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:17.969907 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:17.969860    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:18.519590 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:18.519554    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:21.419226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:21.419196    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:21.646531 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:21.646501    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:23.167963 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:23.167920    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:23.777044 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:23.777010    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:25.283983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:25.283948    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:26.126266 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:26.126234    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:26.459465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:26.459380    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:26.947812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:26.947370    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:29.882464 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:29.882427    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:30.103373 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:30.103341    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:32.364091 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:32.364060    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:33.170637 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:33.170604    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:33.436760 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:33.436638    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:33.665821 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:33.665177    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:33.842377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:33.842291    4900 generic.go:356] "Generic (PLEG): container finished" podID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" containerID="c1cc9234a64c15ae7fdb69bd6013b6720b0921391d19e5d5b39e085a8446b87f" exitCode=0
Oct 31 03:40:33.842527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:33.842365    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs" event={"ID":"0e56ee62-55fe-4c65-98d2-15ef41f72bb5","Type":"ContainerDied","Data":"c1cc9234a64c15ae7fdb69bd6013b6720b0921391d19e5d5b39e085a8446b87f"}
Oct 31 03:40:33.842863 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:33.842848    4900 scope.go:122] "RemoveContainer" containerID="c1cc9234a64c15ae7fdb69bd6013b6720b0921391d19e5d5b39e085a8446b87f"
Oct 31 03:40:33.844449 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:33.844424    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:33.844518 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:33.844492    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:33.844554 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:33.844527    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:34.536245 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:34.536147    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:34.875613 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:34.875481    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs" event={"ID":"0e56ee62-55fe-4c65-98d2-15ef41f72bb5","Type":"ContainerStarted","Data":"f4942df48ca71bca14c3f7e571c96166719bad5a39471cccd8eb18fb1bad1e9d"}
Oct 31 03:40:35.198229 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:35.190625    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:35.501988 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:35.501958    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:35.573427 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:35.573396    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:40:35.573832 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:35.573595    4900 patch_prober.go:29] interesting pod/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 03:40:35.573832 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:35.573643    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs" podUID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 03:40:36.573164 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:36.573129    4900 patch_prober.go:29] interesting pod/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 03:40:36.573356 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:36.573190    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs" podUID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 03:40:36.986582 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:40:36.986548    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:40:37.573381 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:37.573340    4900 patch_prober.go:29] interesting pod/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 03:40:37.573604 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:37.573393    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs" podUID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 03:40:37.590707 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:37.590670    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:38.573547 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:38.573507    4900 patch_prober.go:29] interesting pod/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 03:40:38.574025 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:38.573564    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs" podUID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 03:40:39.573585 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:39.573497    4900 patch_prober.go:29] interesting pod/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 03:40:39.573585 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:39.573555    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs" podUID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 03:40:39.949820 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:39.949696    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:40.175658 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:40.175629    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:40.578125 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:40.578091    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:40:40.578535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:40.578358    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:40:40.579141 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:40.579125    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:40:43.216279 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:43.216247    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:43.423184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:43.423151    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:44.688256 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:44.688218    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:49.309728 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:49.309692    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:49.892553 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:49.892517    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:50.095354 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:50.095313    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:51.432224 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:51.432181    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:53.207993 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:53.207957    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:53.421051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:53.421015    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:54.806531 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:54.806489    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:56.308939 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:56.308908    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:56.880395 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:56.880360    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:57.084891 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:57.084842    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:57.556566 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:57.556532    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:40:57.775262 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:40:57.775221    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:00.567941 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:00.567887    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:00.913386 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:00.913301    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"]
Oct 31 03:41:01.026894 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:01.026859    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"]
Oct 31 03:41:02.390997 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:02.390965    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:02.611540 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:02.611510    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:02.835616 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:02.835577    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:05.913961 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:05.913916    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m" podUID="4b9ab7b8-5f81-4b67-b7c3-1dee661bb544" containerName="payload-processing" containerID="cri-o://62f8957b47962a285b094fa4e5191ecf5758af9a1c5988d80467c6e82e30255c" gracePeriod=25
Oct 31 03:41:06.027496 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.027451    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw" podUID="c931b976-b5fd-4c17-8f77-89a726ff17d8" containerName="payload-pre-processing" containerID="cri-o://2b11d2330155350540f1b3433c7dc538d0063438f29c91e96ccb58565b7d2bb2" gracePeriod=25
Oct 31 03:41:06.073198 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.073164    4900 generic.go:356] "Generic (PLEG): container finished" podID="4b9ab7b8-5f81-4b67-b7c3-1dee661bb544" containerID="62f8957b47962a285b094fa4e5191ecf5758af9a1c5988d80467c6e82e30255c" exitCode=0
Oct 31 03:41:06.073363 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.073235    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m" event={"ID":"4b9ab7b8-5f81-4b67-b7c3-1dee661bb544","Type":"ContainerDied","Data":"62f8957b47962a285b094fa4e5191ecf5758af9a1c5988d80467c6e82e30255c"}
Oct 31 03:41:06.190375 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.190348    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"
Oct 31 03:41:06.242996 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.242962    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544-kube-api-access-2q5rz\" (UniqueName: \"kubernetes.io/projected/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544-kube-api-access-2q5rz\") pod \"4b9ab7b8-5f81-4b67-b7c3-1dee661bb544\" (UID: \"4b9ab7b8-5f81-4b67-b7c3-1dee661bb544\") "
Oct 31 03:41:06.243220 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.243206    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544-plugins-config-volume\") pod \"4b9ab7b8-5f81-4b67-b7c3-1dee661bb544\" (UID: \"4b9ab7b8-5f81-4b67-b7c3-1dee661bb544\") "
Oct 31 03:41:06.243610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.243567    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544-plugins-config-volume" pod "4b9ab7b8-5f81-4b67-b7c3-1dee661bb544" (UID: "4b9ab7b8-5f81-4b67-b7c3-1dee661bb544"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:06.243738 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.243720    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:06.245203 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.245180    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544-kube-api-access-2q5rz" pod "4b9ab7b8-5f81-4b67-b7c3-1dee661bb544" (UID: "4b9ab7b8-5f81-4b67-b7c3-1dee661bb544"). InnerVolumeSpecName "kube-api-access-2q5rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:06.282828 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.282803    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"
Oct 31 03:41:06.344580 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.344547    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c931b976-b5fd-4c17-8f77-89a726ff17d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c931b976-b5fd-4c17-8f77-89a726ff17d8-plugins-config-volume\") pod \"c931b976-b5fd-4c17-8f77-89a726ff17d8\" (UID: \"c931b976-b5fd-4c17-8f77-89a726ff17d8\") "
Oct 31 03:41:06.344773 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.344648    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c931b976-b5fd-4c17-8f77-89a726ff17d8-kube-api-access-c75z8\" (UniqueName: \"kubernetes.io/projected/c931b976-b5fd-4c17-8f77-89a726ff17d8-kube-api-access-c75z8\") pod \"c931b976-b5fd-4c17-8f77-89a726ff17d8\" (UID: \"c931b976-b5fd-4c17-8f77-89a726ff17d8\") "
Oct 31 03:41:06.345027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.345001    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c931b976-b5fd-4c17-8f77-89a726ff17d8-plugins-config-volume" pod "c931b976-b5fd-4c17-8f77-89a726ff17d8" (UID: "c931b976-b5fd-4c17-8f77-89a726ff17d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:06.345277 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.345261    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q5rz\" (UniqueName: \"kubernetes.io/projected/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544-kube-api-access-2q5rz\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:06.345325 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.345279    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c931b976-b5fd-4c17-8f77-89a726ff17d8-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:06.346753 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.346719    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c931b976-b5fd-4c17-8f77-89a726ff17d8-kube-api-access-c75z8" pod "c931b976-b5fd-4c17-8f77-89a726ff17d8" (UID: "c931b976-b5fd-4c17-8f77-89a726ff17d8"). InnerVolumeSpecName "kube-api-access-c75z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:06.447218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.447126    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c75z8\" (UniqueName: \"kubernetes.io/projected/c931b976-b5fd-4c17-8f77-89a726ff17d8-kube-api-access-c75z8\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:06.793273 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:06.793241    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:06.998559 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:41:06.998517    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:41:07.010200 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.008859    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:07.084278 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.084179    4900 generic.go:356] "Generic (PLEG): container finished" podID="c931b976-b5fd-4c17-8f77-89a726ff17d8" containerID="2b11d2330155350540f1b3433c7dc538d0063438f29c91e96ccb58565b7d2bb2" exitCode=0
Oct 31 03:41:07.084278 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.084221    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"
Oct 31 03:41:07.084278 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.084247    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw" event={"ID":"c931b976-b5fd-4c17-8f77-89a726ff17d8","Type":"ContainerDied","Data":"2b11d2330155350540f1b3433c7dc538d0063438f29c91e96ccb58565b7d2bb2"}
Oct 31 03:41:07.084558 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.084304    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw" event={"ID":"c931b976-b5fd-4c17-8f77-89a726ff17d8","Type":"ContainerDied","Data":"c91065c241393b711e6a9e742c4f3e5b4d9a5301da4d8d38a6316f3518ec8062"}
Oct 31 03:41:07.084558 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.084328    4900 scope.go:122] "RemoveContainer" containerID="2b11d2330155350540f1b3433c7dc538d0063438f29c91e96ccb58565b7d2bb2"
Oct 31 03:41:07.086650 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.086623    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m" event={"ID":"4b9ab7b8-5f81-4b67-b7c3-1dee661bb544","Type":"ContainerDied","Data":"87383da9d0da6f728d36a0c1b2b9193e592c31f11aa33168b549203e8c904db2"}
Oct 31 03:41:07.086896 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.086876    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"
Oct 31 03:41:07.099748 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.099726    4900 scope.go:122] "RemoveContainer" containerID="2b11d2330155350540f1b3433c7dc538d0063438f29c91e96ccb58565b7d2bb2"
Oct 31 03:41:07.100213 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:41:07.100187    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b11d2330155350540f1b3433c7dc538d0063438f29c91e96ccb58565b7d2bb2\": container with ID starting with 2b11d2330155350540f1b3433c7dc538d0063438f29c91e96ccb58565b7d2bb2 not found: ID does not exist" containerID="2b11d2330155350540f1b3433c7dc538d0063438f29c91e96ccb58565b7d2bb2"
Oct 31 03:41:07.100307 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.100224    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b11d2330155350540f1b3433c7dc538d0063438f29c91e96ccb58565b7d2bb2"} err="failed to get container status \"2b11d2330155350540f1b3433c7dc538d0063438f29c91e96ccb58565b7d2bb2\": rpc error: code = NotFound desc = could not find container \"2b11d2330155350540f1b3433c7dc538d0063438f29c91e96ccb58565b7d2bb2\": container with ID starting with 2b11d2330155350540f1b3433c7dc538d0063438f29c91e96ccb58565b7d2bb2 not found: ID does not exist"
Oct 31 03:41:07.100307 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.100251    4900 scope.go:122] "RemoveContainer" containerID="62f8957b47962a285b094fa4e5191ecf5758af9a1c5988d80467c6e82e30255c"
Oct 31 03:41:07.258849 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.255997    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"]
Oct 31 03:41:07.258849 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.257690    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c35264-648bf5c4c-pjvnw"]
Oct 31 03:41:07.358624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.358527    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"]
Oct 31 03:41:07.362580 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.362548    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c35264-74c6497b4f-wmc9m"]
Oct 31 03:41:07.956752 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:07.956716    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:08.540334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:08.540274    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:09.154048 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:09.154016    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b9ab7b8-5f81-4b67-b7c3-1dee661bb544" path="/var/lib/kubelet/pods/4b9ab7b8-5f81-4b67-b7c3-1dee661bb544/volumes"
Oct 31 03:41:09.154452 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:09.154439    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c931b976-b5fd-4c17-8f77-89a726ff17d8" path="/var/lib/kubelet/pods/c931b976-b5fd-4c17-8f77-89a726ff17d8/volumes"
Oct 31 03:41:13.221941 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:13.221878    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:13.431581 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:13.431537    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:13.639638 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:13.639597    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:18.604976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:18.604902    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:25.187878 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:25.187835    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:28.124761 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:28.124720    4900 scope.go:122] "RemoveContainer" containerID="6c214fed12307f1e6eef206162a023161df5f51957fba56db9318a570ffc67d7"
Oct 31 03:41:30.149108 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:30.149074    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:32.784055 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:32.784023    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:32.988994 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:32.988961    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:35.047807 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:35.046150    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:35.279450 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:35.279409    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:36.998536 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:41:36.998488    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:41:40.183194 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:40.183161    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:43.987099 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:43.987041    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:44.546521 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:44.546485    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:44.893455 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:44.893369    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"]
Oct 31 03:41:44.893719 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:44.893691    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4" podUID="4b41ee43-543f-4e36-8b24-ca770073064c" containerName="payload-processing" containerID="cri-o://323fc83703c39f9efc966f9bbdfd050b6763f53ab775bd7c027a3850dbd8ec41" gracePeriod=30
Oct 31 03:41:45.132568 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.132535    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:45.192877 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.192848    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"
Oct 31 03:41:45.333024 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.332979    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:45.347727 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.347697    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b41ee43-543f-4e36-8b24-ca770073064c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b41ee43-543f-4e36-8b24-ca770073064c-plugins-config-volume\") pod \"4b41ee43-543f-4e36-8b24-ca770073064c\" (UID: \"4b41ee43-543f-4e36-8b24-ca770073064c\") "
Oct 31 03:41:45.348175 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.347743    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b41ee43-543f-4e36-8b24-ca770073064c-kube-api-access-gk9td\" (UniqueName: \"kubernetes.io/projected/4b41ee43-543f-4e36-8b24-ca770073064c-kube-api-access-gk9td\") pod \"4b41ee43-543f-4e36-8b24-ca770073064c\" (UID: \"4b41ee43-543f-4e36-8b24-ca770073064c\") "
Oct 31 03:41:45.348175 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.348103    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b41ee43-543f-4e36-8b24-ca770073064c-plugins-config-volume" pod "4b41ee43-543f-4e36-8b24-ca770073064c" (UID: "4b41ee43-543f-4e36-8b24-ca770073064c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:45.348374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.348355    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b41ee43-543f-4e36-8b24-ca770073064c-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:45.349954 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.349929    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b41ee43-543f-4e36-8b24-ca770073064c-kube-api-access-gk9td" pod "4b41ee43-543f-4e36-8b24-ca770073064c" (UID: "4b41ee43-543f-4e36-8b24-ca770073064c"). InnerVolumeSpecName "kube-api-access-gk9td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:45.357623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.357597    4900 generic.go:356] "Generic (PLEG): container finished" podID="4b41ee43-543f-4e36-8b24-ca770073064c" containerID="323fc83703c39f9efc966f9bbdfd050b6763f53ab775bd7c027a3850dbd8ec41" exitCode=0
Oct 31 03:41:45.357759 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.357642    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"
Oct 31 03:41:45.357759 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.357675    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4" event={"ID":"4b41ee43-543f-4e36-8b24-ca770073064c","Type":"ContainerDied","Data":"323fc83703c39f9efc966f9bbdfd050b6763f53ab775bd7c027a3850dbd8ec41"}
Oct 31 03:41:45.357759 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.357738    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4" event={"ID":"4b41ee43-543f-4e36-8b24-ca770073064c","Type":"ContainerDied","Data":"9cde85f69b5a70d74b8603ac3f3c24a39ad28a5fd6a39c86a663db98896bf7ef"}
Oct 31 03:41:45.357894 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.357770    4900 scope.go:122] "RemoveContainer" containerID="323fc83703c39f9efc966f9bbdfd050b6763f53ab775bd7c027a3850dbd8ec41"
Oct 31 03:41:45.367942 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.367923    4900 scope.go:122] "RemoveContainer" containerID="323fc83703c39f9efc966f9bbdfd050b6763f53ab775bd7c027a3850dbd8ec41"
Oct 31 03:41:45.368223 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:41:45.368201    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323fc83703c39f9efc966f9bbdfd050b6763f53ab775bd7c027a3850dbd8ec41\": container with ID starting with 323fc83703c39f9efc966f9bbdfd050b6763f53ab775bd7c027a3850dbd8ec41 not found: ID does not exist" containerID="323fc83703c39f9efc966f9bbdfd050b6763f53ab775bd7c027a3850dbd8ec41"
Oct 31 03:41:45.368269 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.368233    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323fc83703c39f9efc966f9bbdfd050b6763f53ab775bd7c027a3850dbd8ec41"} err="failed to get container status \"323fc83703c39f9efc966f9bbdfd050b6763f53ab775bd7c027a3850dbd8ec41\": rpc error: code = NotFound desc = could not find container \"323fc83703c39f9efc966f9bbdfd050b6763f53ab775bd7c027a3850dbd8ec41\": container with ID starting with 323fc83703c39f9efc966f9bbdfd050b6763f53ab775bd7c027a3850dbd8ec41 not found: ID does not exist"
Oct 31 03:41:45.453224 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.453115    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk9td\" (UniqueName: \"kubernetes.io/projected/4b41ee43-543f-4e36-8b24-ca770073064c-kube-api-access-gk9td\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:45.518890 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.518851    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"]
Oct 31 03:41:45.519174 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:45.519150    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8b4030-fcbf99c6d-jtxq4"]
Oct 31 03:41:46.404680 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:46.404625    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:46.609421 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:46.609391    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:46.810635 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:46.810605    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:47.153384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:47.153307    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b41ee43-543f-4e36-8b24-ca770073064c" path="/var/lib/kubelet/pods/4b41ee43-543f-4e36-8b24-ca770073064c/volumes"
Oct 31 03:41:50.266011 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:50.265972    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:51.965467 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:51.965430    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:52.411766 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:52.411694    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:53.222127 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:53.222097    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:53.751254 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:53.751223    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:57.223932 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:57.223904    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:41:59.184765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:41:59.184697    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:01.247252 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:01.247217    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:01.756076 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:01.756037    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:04.728220 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:04.728182    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:06.984341 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:42:06.984300    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:42:10.113293 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:10.113190    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:10.312627 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:10.312594    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:10.516614 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:10.516581    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:15.273679 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:15.273637    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:20.307387 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:20.307346    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:25.191000 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:25.190958    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:28.572400 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:28.572373    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:42:28.615809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:28.615761    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:42:30.375090 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:30.375053    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:35.126416 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:35.126387    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:36.984233 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:42:36.984194    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:42:37.600074 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:37.600030    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:37.805983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:37.805951    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:42.598841 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:42.598809    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:43.397475 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:43.397415    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:44.070570 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:44.070537    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"]
Oct 31 03:42:44.524758 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:44.524710    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:44.786757 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:44.786672    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:47.136920 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:47.136885    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"]
Oct 31 03:42:47.137408 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:47.137122    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2" podUID="aaf0d481-00ac-4262-8bc7-224f1ff8f54b" containerName="istio-proxy" containerID="cri-o://bfdfc0c74e76f95bae2fc4eaa25f2d30eed5a1ee18bdf7d835834bf74c2f1466" gracePeriod=30
Oct 31 03:42:47.183201 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:47.183164    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"]
Oct 31 03:42:47.183412 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:47.183389    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" podUID="baed435c-cff8-4c33-b642-a1fd39d437f9" containerName="istio-proxy" containerID="cri-o://45d0fc2c1bae0e7bf8bf05e21b0e9a822c7a4dd01a6c330d381c143e8fcffa7d" gracePeriod=30
Oct 31 03:42:47.972564 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:47.972526    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:48.886000 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:48.885969    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:49.071047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.071005    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp" podUID="777d67a4-f930-450a-acac-b1b933ceada0" containerName="payload-processing" containerID="cri-o://0b6ec974accf4ce3e4a27982b25c830386ae0d4b97075e20d012c2c0c8698541" gracePeriod=25
Oct 31 03:42:49.147452 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.147379    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-pflrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:42:49.147583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.147448    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" podUID="baed435c-cff8-4c33-b642-a1fd39d437f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:42:49.337198 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.337172    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"
Oct 31 03:42:49.379429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.379400    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777d67a4-f930-450a-acac-b1b933ceada0-kube-api-access-gpktw\" (UniqueName: \"kubernetes.io/projected/777d67a4-f930-450a-acac-b1b933ceada0-kube-api-access-gpktw\") pod \"777d67a4-f930-450a-acac-b1b933ceada0\" (UID: \"777d67a4-f930-450a-acac-b1b933ceada0\") "
Oct 31 03:42:49.379610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.379443    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777d67a4-f930-450a-acac-b1b933ceada0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777d67a4-f930-450a-acac-b1b933ceada0-plugins-config-volume\") pod \"777d67a4-f930-450a-acac-b1b933ceada0\" (UID: \"777d67a4-f930-450a-acac-b1b933ceada0\") "
Oct 31 03:42:49.379929 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.379899    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777d67a4-f930-450a-acac-b1b933ceada0-plugins-config-volume" pod "777d67a4-f930-450a-acac-b1b933ceada0" (UID: "777d67a4-f930-450a-acac-b1b933ceada0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:42:49.381831 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.381758    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777d67a4-f930-450a-acac-b1b933ceada0-kube-api-access-gpktw" pod "777d67a4-f930-450a-acac-b1b933ceada0" (UID: "777d67a4-f930-450a-acac-b1b933ceada0"). InnerVolumeSpecName "kube-api-access-gpktw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:49.481472 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.481387    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpktw\" (UniqueName: \"kubernetes.io/projected/777d67a4-f930-450a-acac-b1b933ceada0-kube-api-access-gpktw\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:49.481472 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.481418    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/777d67a4-f930-450a-acac-b1b933ceada0-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:49.741798 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.741757    4900 generic.go:356] "Generic (PLEG): container finished" podID="777d67a4-f930-450a-acac-b1b933ceada0" containerID="0b6ec974accf4ce3e4a27982b25c830386ae0d4b97075e20d012c2c0c8698541" exitCode=0
Oct 31 03:42:49.741975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.741843    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"
Oct 31 03:42:49.741975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.741852    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp" event={"ID":"777d67a4-f930-450a-acac-b1b933ceada0","Type":"ContainerDied","Data":"0b6ec974accf4ce3e4a27982b25c830386ae0d4b97075e20d012c2c0c8698541"}
Oct 31 03:42:49.741975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.741914    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp" event={"ID":"777d67a4-f930-450a-acac-b1b933ceada0","Type":"ContainerDied","Data":"ea0d39e3b84bad5fb3ff46a8cae03e3f03a42b264c84faec87a5234b9bd414ff"}
Oct 31 03:42:49.742082 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.741977    4900 scope.go:122] "RemoveContainer" containerID="0b6ec974accf4ce3e4a27982b25c830386ae0d4b97075e20d012c2c0c8698541"
Oct 31 03:42:49.751563 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.751542    4900 scope.go:122] "RemoveContainer" containerID="0b6ec974accf4ce3e4a27982b25c830386ae0d4b97075e20d012c2c0c8698541"
Oct 31 03:42:49.751859 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:42:49.751837    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b6ec974accf4ce3e4a27982b25c830386ae0d4b97075e20d012c2c0c8698541\": container with ID starting with 0b6ec974accf4ce3e4a27982b25c830386ae0d4b97075e20d012c2c0c8698541 not found: ID does not exist" containerID="0b6ec974accf4ce3e4a27982b25c830386ae0d4b97075e20d012c2c0c8698541"
Oct 31 03:42:49.751918 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.751868    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b6ec974accf4ce3e4a27982b25c830386ae0d4b97075e20d012c2c0c8698541"} err="failed to get container status \"0b6ec974accf4ce3e4a27982b25c830386ae0d4b97075e20d012c2c0c8698541\": rpc error: code = NotFound desc = could not find container \"0b6ec974accf4ce3e4a27982b25c830386ae0d4b97075e20d012c2c0c8698541\": container with ID starting with 0b6ec974accf4ce3e4a27982b25c830386ae0d4b97075e20d012c2c0c8698541 not found: ID does not exist"
Oct 31 03:42:49.898467 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.898433    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"]
Oct 31 03:42:49.944607 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:49.944572    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-329d79-5b55f4444c-xm7lp"]
Oct 31 03:42:51.153124 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:51.153086    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777d67a4-f930-450a-acac-b1b933ceada0" path="/var/lib/kubelet/pods/777d67a4-f930-450a-acac-b1b933ceada0/volumes"
Oct 31 03:42:52.514019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.513992    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:42:52.528923 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.528901    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:42:52.620327 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620302    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-workload-certs\") pod \"baed435c-cff8-4c33-b642-a1fd39d437f9\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") "
Oct 31 03:42:52.620488 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620388    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-envoy\") pod \"baed435c-cff8-4c33-b642-a1fd39d437f9\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") "
Oct 31 03:42:52.620488 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620457    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istiod-ca-cert\") pod \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") "
Oct 31 03:42:52.620607 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620492    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-credential-socket\") pod \"baed435c-cff8-4c33-b642-a1fd39d437f9\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") "
Oct 31 03:42:52.620607 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620531    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-workload-certs\") pod \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") "
Oct 31 03:42:52.620607 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620557    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-workload-certs" pod "baed435c-cff8-4c33-b642-a1fd39d437f9" (UID: "baed435c-cff8-4c33-b642-a1fd39d437f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:52.620607 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620564    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-token\" (UniqueName: \"kubernetes.io/projected/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-token\") pod \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") "
Oct 31 03:42:52.620928 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620659    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-workload-socket\") pod \"baed435c-cff8-4c33-b642-a1fd39d437f9\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") "
Oct 31 03:42:52.620928 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620730    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-workload-socket\") pod \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") "
Oct 31 03:42:52.620928 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620759    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-credential-socket" pod "baed435c-cff8-4c33-b642-a1fd39d437f9" (UID: "baed435c-cff8-4c33-b642-a1fd39d437f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:52.620928 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620773    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-kube-api-access-vj8lc\" (UniqueName: \"kubernetes.io/projected/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-kube-api-access-vj8lc\") pod \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") "
Oct 31 03:42:52.620928 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620848    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-podinfo\") pod \"baed435c-cff8-4c33-b642-a1fd39d437f9\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") "
Oct 31 03:42:52.620928 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620916    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-credential-socket\") pod \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") "
Oct 31 03:42:52.621249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620898    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-workload-certs" pod "aaf0d481-00ac-4262-8bc7-224f1ff8f54b" (UID: "aaf0d481-00ac-4262-8bc7-224f1ff8f54b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:52.621249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.620988    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-data\") pod \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") "
Oct 31 03:42:52.621249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621000    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istiod-ca-cert" pod "aaf0d481-00ac-4262-8bc7-224f1ff8f54b" (UID: "aaf0d481-00ac-4262-8bc7-224f1ff8f54b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:42:52.621249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621003    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-workload-socket" pod "aaf0d481-00ac-4262-8bc7-224f1ff8f54b" (UID: "aaf0d481-00ac-4262-8bc7-224f1ff8f54b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:52.621249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621017    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-envoy\") pod \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") "
Oct 31 03:42:52.621249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621066    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baed435c-cff8-4c33-b642-a1fd39d437f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baed435c-cff8-4c33-b642-a1fd39d437f9-istiod-ca-cert\") pod \"baed435c-cff8-4c33-b642-a1fd39d437f9\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") "
Oct 31 03:42:52.621249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621124    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-token\" (UniqueName: \"kubernetes.io/projected/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-token\") pod \"baed435c-cff8-4c33-b642-a1fd39d437f9\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") "
Oct 31 03:42:52.621249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621182    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-data\") pod \"baed435c-cff8-4c33-b642-a1fd39d437f9\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") "
Oct 31 03:42:52.621249 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621210    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-credential-socket" pod "aaf0d481-00ac-4262-8bc7-224f1ff8f54b" (UID: "aaf0d481-00ac-4262-8bc7-224f1ff8f54b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:52.621725 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621251    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baed435c-cff8-4c33-b642-a1fd39d437f9-kube-api-access-8qf9m\" (UniqueName: \"kubernetes.io/projected/baed435c-cff8-4c33-b642-a1fd39d437f9-kube-api-access-8qf9m\") pod \"baed435c-cff8-4c33-b642-a1fd39d437f9\" (UID: \"baed435c-cff8-4c33-b642-a1fd39d437f9\") "
Oct 31 03:42:52.621725 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621285    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-podinfo\") pod \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\" (UID: \"aaf0d481-00ac-4262-8bc7-224f1ff8f54b\") "
Oct 31 03:42:52.621725 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621397    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-data" pod "aaf0d481-00ac-4262-8bc7-224f1ff8f54b" (UID: "aaf0d481-00ac-4262-8bc7-224f1ff8f54b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:52.621725 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621602    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-data" pod "baed435c-cff8-4c33-b642-a1fd39d437f9" (UID: "baed435c-cff8-4c33-b642-a1fd39d437f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:52.621725 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621602    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baed435c-cff8-4c33-b642-a1fd39d437f9-istiod-ca-cert" pod "baed435c-cff8-4c33-b642-a1fd39d437f9" (UID: "baed435c-cff8-4c33-b642-a1fd39d437f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:42:52.622007 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621919    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.622007 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621941    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.622007 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621956    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.622007 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621971    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.622007 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621985    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.622007 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.621998    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.622315 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.622012    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.622315 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.622025    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/baed435c-cff8-4c33-b642-a1fd39d437f9-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.622315 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.622039    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.622315 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.622120    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-workload-socket" pod "baed435c-cff8-4c33-b642-a1fd39d437f9" (UID: "baed435c-cff8-4c33-b642-a1fd39d437f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:52.623133 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.623107    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-token" pod "aaf0d481-00ac-4262-8bc7-224f1ff8f54b" (UID: "aaf0d481-00ac-4262-8bc7-224f1ff8f54b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:52.623250 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.623185    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-envoy" pod "baed435c-cff8-4c33-b642-a1fd39d437f9" (UID: "baed435c-cff8-4c33-b642-a1fd39d437f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:52.623384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.623360    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-kube-api-access-vj8lc" pod "aaf0d481-00ac-4262-8bc7-224f1ff8f54b" (UID: "aaf0d481-00ac-4262-8bc7-224f1ff8f54b"). InnerVolumeSpecName "kube-api-access-vj8lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:52.623737 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.623712    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-envoy" pod "aaf0d481-00ac-4262-8bc7-224f1ff8f54b" (UID: "aaf0d481-00ac-4262-8bc7-224f1ff8f54b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:52.623838 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.623743    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baed435c-cff8-4c33-b642-a1fd39d437f9-kube-api-access-8qf9m" pod "baed435c-cff8-4c33-b642-a1fd39d437f9" (UID: "baed435c-cff8-4c33-b642-a1fd39d437f9"). InnerVolumeSpecName "kube-api-access-8qf9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:52.624118 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.624102    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-podinfo" pod "baed435c-cff8-4c33-b642-a1fd39d437f9" (UID: "baed435c-cff8-4c33-b642-a1fd39d437f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:42:52.624192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.624177    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-podinfo" pod "aaf0d481-00ac-4262-8bc7-224f1ff8f54b" (UID: "aaf0d481-00ac-4262-8bc7-224f1ff8f54b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:42:52.624501 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.624484    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-token" pod "baed435c-cff8-4c33-b642-a1fd39d437f9" (UID: "baed435c-cff8-4c33-b642-a1fd39d437f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:52.723595 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.723565    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.723595 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.723592    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.723801 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.723604    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj8lc\" (UniqueName: \"kubernetes.io/projected/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-kube-api-access-vj8lc\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.723801 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.723615    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.723801 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.723623    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.723801 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.723631    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.723801 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.723639    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qf9m\" (UniqueName: \"kubernetes.io/projected/baed435c-cff8-4c33-b642-a1fd39d437f9-kube-api-access-8qf9m\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.723801 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.723649    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf0d481-00ac-4262-8bc7-224f1ff8f54b-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.723801 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.723657    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/baed435c-cff8-4c33-b642-a1fd39d437f9-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:52.760520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.760446    4900 generic.go:356] "Generic (PLEG): container finished" podID="aaf0d481-00ac-4262-8bc7-224f1ff8f54b" containerID="bfdfc0c74e76f95bae2fc4eaa25f2d30eed5a1ee18bdf7d835834bf74c2f1466" exitCode=0
Oct 31 03:42:52.760520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.760495    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"
Oct 31 03:42:52.760706 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.760530    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2" event={"ID":"aaf0d481-00ac-4262-8bc7-224f1ff8f54b","Type":"ContainerDied","Data":"bfdfc0c74e76f95bae2fc4eaa25f2d30eed5a1ee18bdf7d835834bf74c2f1466"}
Oct 31 03:42:52.760706 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.760599    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2" event={"ID":"aaf0d481-00ac-4262-8bc7-224f1ff8f54b","Type":"ContainerDied","Data":"7f38c06e8ecb49f050ebfaf9d9f0ab140d68f83833f29b6467943503f6d0167e"}
Oct 31 03:42:52.760706 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.760621    4900 scope.go:122] "RemoveContainer" containerID="bfdfc0c74e76f95bae2fc4eaa25f2d30eed5a1ee18bdf7d835834bf74c2f1466"
Oct 31 03:42:52.762066 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.762044    4900 generic.go:356] "Generic (PLEG): container finished" podID="baed435c-cff8-4c33-b642-a1fd39d437f9" containerID="45d0fc2c1bae0e7bf8bf05e21b0e9a822c7a4dd01a6c330d381c143e8fcffa7d" exitCode=0
Oct 31 03:42:52.762177 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.762089    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"
Oct 31 03:42:52.762177 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.762098    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" event={"ID":"baed435c-cff8-4c33-b642-a1fd39d437f9","Type":"ContainerDied","Data":"45d0fc2c1bae0e7bf8bf05e21b0e9a822c7a4dd01a6c330d381c143e8fcffa7d"}
Oct 31 03:42:52.762177 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.762130    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt" event={"ID":"baed435c-cff8-4c33-b642-a1fd39d437f9","Type":"ContainerDied","Data":"b8ed34d86cf92d2ee5847d7204b527ddb0c92429e7ef338c27c7b5a65afabc83"}
Oct 31 03:42:52.770478 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.770459    4900 scope.go:122] "RemoveContainer" containerID="bfdfc0c74e76f95bae2fc4eaa25f2d30eed5a1ee18bdf7d835834bf74c2f1466"
Oct 31 03:42:52.770809 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:42:52.770766    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfdfc0c74e76f95bae2fc4eaa25f2d30eed5a1ee18bdf7d835834bf74c2f1466\": container with ID starting with bfdfc0c74e76f95bae2fc4eaa25f2d30eed5a1ee18bdf7d835834bf74c2f1466 not found: ID does not exist" containerID="bfdfc0c74e76f95bae2fc4eaa25f2d30eed5a1ee18bdf7d835834bf74c2f1466"
Oct 31 03:42:52.770893 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.770811    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfdfc0c74e76f95bae2fc4eaa25f2d30eed5a1ee18bdf7d835834bf74c2f1466"} err="failed to get container status \"bfdfc0c74e76f95bae2fc4eaa25f2d30eed5a1ee18bdf7d835834bf74c2f1466\": rpc error: code = NotFound desc = could not find container \"bfdfc0c74e76f95bae2fc4eaa25f2d30eed5a1ee18bdf7d835834bf74c2f1466\": container with ID starting with bfdfc0c74e76f95bae2fc4eaa25f2d30eed5a1ee18bdf7d835834bf74c2f1466 not found: ID does not exist"
Oct 31 03:42:52.770893 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.770834    4900 scope.go:122] "RemoveContainer" containerID="45d0fc2c1bae0e7bf8bf05e21b0e9a822c7a4dd01a6c330d381c143e8fcffa7d"
Oct 31 03:42:52.780374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.780164    4900 scope.go:122] "RemoveContainer" containerID="45d0fc2c1bae0e7bf8bf05e21b0e9a822c7a4dd01a6c330d381c143e8fcffa7d"
Oct 31 03:42:52.780687 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:42:52.780658    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d0fc2c1bae0e7bf8bf05e21b0e9a822c7a4dd01a6c330d381c143e8fcffa7d\": container with ID starting with 45d0fc2c1bae0e7bf8bf05e21b0e9a822c7a4dd01a6c330d381c143e8fcffa7d not found: ID does not exist" containerID="45d0fc2c1bae0e7bf8bf05e21b0e9a822c7a4dd01a6c330d381c143e8fcffa7d"
Oct 31 03:42:52.780769 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.780691    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d0fc2c1bae0e7bf8bf05e21b0e9a822c7a4dd01a6c330d381c143e8fcffa7d"} err="failed to get container status \"45d0fc2c1bae0e7bf8bf05e21b0e9a822c7a4dd01a6c330d381c143e8fcffa7d\": rpc error: code = NotFound desc = could not find container \"45d0fc2c1bae0e7bf8bf05e21b0e9a822c7a4dd01a6c330d381c143e8fcffa7d\": container with ID starting with 45d0fc2c1bae0e7bf8bf05e21b0e9a822c7a4dd01a6c330d381c143e8fcffa7d not found: ID does not exist"
Oct 31 03:42:52.884316 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.884284    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"]
Oct 31 03:42:52.891347 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.891318    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-dffz2"]
Oct 31 03:42:52.991989 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.991933    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"]
Oct 31 03:42:52.995592 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:52.995567    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pflrt"]
Oct 31 03:42:53.155435 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:53.155354    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf0d481-00ac-4262-8bc7-224f1ff8f54b" path="/var/lib/kubelet/pods/aaf0d481-00ac-4262-8bc7-224f1ff8f54b/volumes"
Oct 31 03:42:53.155896 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:53.155881    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baed435c-cff8-4c33-b642-a1fd39d437f9" path="/var/lib/kubelet/pods/baed435c-cff8-4c33-b642-a1fd39d437f9/volumes"
Oct 31 03:42:53.172827 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:53.167914    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:54.118319 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:54.118278    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:54.382097 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:54.382003    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:56.251879 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:56.251840    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"]
Oct 31 03:42:56.252343 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:56.252065    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f" podUID="d698f07e-e83e-4b52-9aba-1699cf8d1841" containerName="istio-proxy" containerID="cri-o://d2e4a7aab6d453d49f946247b0551e87244d7fe796e2fe16545fa08e37980bf6" gracePeriod=30
Oct 31 03:42:57.774734 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:57.774698    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:42:57.775137 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:57.774762    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f" podUID="d698f07e-e83e-4b52-9aba-1699cf8d1841" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:42:58.415510 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:58.415479    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:58.640717 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:58.640682    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:42:59.776937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:42:59.776902    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:00.274075 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274041    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"]
Oct 31 03:43:00.274592 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274577    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c931b976-b5fd-4c17-8f77-89a726ff17d8" containerName="payload-pre-processing"
Oct 31 03:43:00.274639 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274594    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c931b976-b5fd-4c17-8f77-89a726ff17d8" containerName="payload-pre-processing"
Oct 31 03:43:00.274639 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274617    4900 cpu_manager.go:414] "containerMap: removing container" podUID="baed435c-cff8-4c33-b642-a1fd39d437f9" containerName="istio-proxy"
Oct 31 03:43:00.274639 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274622    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baed435c-cff8-4c33-b642-a1fd39d437f9" containerName="istio-proxy"
Oct 31 03:43:00.274733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274637    4900 cpu_manager.go:414] "containerMap: removing container" podUID="4b9ab7b8-5f81-4b67-b7c3-1dee661bb544" containerName="payload-processing"
Oct 31 03:43:00.274733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274647    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b9ab7b8-5f81-4b67-b7c3-1dee661bb544" containerName="payload-processing"
Oct 31 03:43:00.274733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274663    4900 cpu_manager.go:414] "containerMap: removing container" podUID="4b41ee43-543f-4e36-8b24-ca770073064c" containerName="payload-processing"
Oct 31 03:43:00.274733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274667    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b41ee43-543f-4e36-8b24-ca770073064c" containerName="payload-processing"
Oct 31 03:43:00.274733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274674    4900 cpu_manager.go:414] "containerMap: removing container" podUID="7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" containerName="istio-proxy"
Oct 31 03:43:00.274733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274678    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" containerName="istio-proxy"
Oct 31 03:43:00.274733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274686    4900 cpu_manager.go:414] "containerMap: removing container" podUID="777d67a4-f930-450a-acac-b1b933ceada0" containerName="payload-processing"
Oct 31 03:43:00.274733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274691    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777d67a4-f930-450a-acac-b1b933ceada0" containerName="payload-processing"
Oct 31 03:43:00.274733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274710    4900 cpu_manager.go:414] "containerMap: removing container" podUID="aaf0d481-00ac-4262-8bc7-224f1ff8f54b" containerName="istio-proxy"
Oct 31 03:43:00.274733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274718    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf0d481-00ac-4262-8bc7-224f1ff8f54b" containerName="istio-proxy"
Oct 31 03:43:00.275051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274809    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf0d481-00ac-4262-8bc7-224f1ff8f54b" containerName="istio-proxy"
Oct 31 03:43:00.275051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274818    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c931b976-b5fd-4c17-8f77-89a726ff17d8" containerName="payload-pre-processing"
Oct 31 03:43:00.275051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274828    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="baed435c-cff8-4c33-b642-a1fd39d437f9" containerName="istio-proxy"
Oct 31 03:43:00.275051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274834    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b9ab7b8-5f81-4b67-b7c3-1dee661bb544" containerName="payload-processing"
Oct 31 03:43:00.275051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274841    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b41ee43-543f-4e36-8b24-ca770073064c" containerName="payload-processing"
Oct 31 03:43:00.275051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274847    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="7745c0a4-53b5-40f9-9b78-b4d510f0c6c5" containerName="istio-proxy"
Oct 31 03:43:00.275051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.274853    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="777d67a4-f930-450a-acac-b1b933ceada0" containerName="payload-processing"
Oct 31 03:43:00.279679 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.279664    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.340909 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.340872    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-03d737-openshift-default-dockercfg-l98r2\""
Oct 31 03:43:00.392287 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.392253    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"]
Oct 31 03:43:00.421978 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.421935    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-workload-certs\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.422145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.422103    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f07fc2e-b24b-46c6-9571-39282e304a53-istiod-ca-cert\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.422191 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.422163    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-data\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.422235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.422201    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-envoy\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.422286 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.422272    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-podinfo\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.422334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.422307    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l282z\" (UniqueName: \"kubernetes.io/projected/5f07fc2e-b24b-46c6-9571-39282e304a53-kube-api-access-l282z\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.422384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.422343    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-token\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.422384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.422373    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-credential-socket\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.422507 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.422429    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-workload-socket\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.450922 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.450895    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:00.523753 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.523714    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-data\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.523753 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.523754    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-envoy\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.523991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.523837    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-podinfo\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.523991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.523873    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l282z\" (UniqueName: \"kubernetes.io/projected/5f07fc2e-b24b-46c6-9571-39282e304a53-kube-api-access-l282z\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.523991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.523914    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-token\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.523991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.523939    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-credential-socket\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.523991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.523988    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-workload-socket\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.524326 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.524048    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-workload-certs\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.524326 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.524145    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f07fc2e-b24b-46c6-9571-39282e304a53-istiod-ca-cert\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.524326 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.524195    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-data\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.524326 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.524313    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-credential-socket\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.524517 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.524402    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-workload-socket\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.524517 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.524464    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-workload-certs\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.524825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.524727    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f07fc2e-b24b-46c6-9571-39282e304a53-istiod-ca-cert\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.526055 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.526036    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-envoy\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.526309 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.526292    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-podinfo\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.579671 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.579633    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l282z\" (UniqueName: \"kubernetes.io/projected/5f07fc2e-b24b-46c6-9571-39282e304a53-kube-api-access-l282z\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.579880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.579635    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-token\") pod \"e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") " pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.592516 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.592488    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:00.662444 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:00.661127    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:01.006494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.006467    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"]
Oct 31 03:43:01.007387 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:43:01.007356    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f07fc2e_b24b_46c6_9571_39282e304a53.slice/crio-63e0ff25d0f09b3d6a0b2447871f1c68f875c4aca6911c992a00a567f4ce4f9a WatchSource:0}: Error finding container 63e0ff25d0f09b3d6a0b2447871f1c68f875c4aca6911c992a00a567f4ce4f9a: Status 404 returned error can't find the container with id 63e0ff25d0f09b3d6a0b2447871f1c68f875c4aca6911c992a00a567f4ce4f9a
Oct 31 03:43:01.009945 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.009915    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:43:01.010051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.009980    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:43:01.010051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.010008    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:43:01.485357 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.485332    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:43:01.637639 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.637610    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-data\") pod \"d698f07e-e83e-4b52-9aba-1699cf8d1841\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") "
Oct 31 03:43:01.637825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.637660    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-workload-socket\") pod \"d698f07e-e83e-4b52-9aba-1699cf8d1841\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") "
Oct 31 03:43:01.637825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.637695    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-podinfo\") pod \"d698f07e-e83e-4b52-9aba-1699cf8d1841\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") "
Oct 31 03:43:01.637825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.637733    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-token\" (UniqueName: \"kubernetes.io/projected/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-token\") pod \"d698f07e-e83e-4b52-9aba-1699cf8d1841\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") "
Oct 31 03:43:01.637825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.637776    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-workload-certs\") pod \"d698f07e-e83e-4b52-9aba-1699cf8d1841\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") "
Oct 31 03:43:01.638084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.637885    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d698f07e-e83e-4b52-9aba-1699cf8d1841-kube-api-access-7tdfk\" (UniqueName: \"kubernetes.io/projected/d698f07e-e83e-4b52-9aba-1699cf8d1841-kube-api-access-7tdfk\") pod \"d698f07e-e83e-4b52-9aba-1699cf8d1841\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") "
Oct 31 03:43:01.638084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.637919    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d698f07e-e83e-4b52-9aba-1699cf8d1841-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d698f07e-e83e-4b52-9aba-1699cf8d1841-istiod-ca-cert\") pod \"d698f07e-e83e-4b52-9aba-1699cf8d1841\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") "
Oct 31 03:43:01.638084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.637954    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-credential-socket\") pod \"d698f07e-e83e-4b52-9aba-1699cf8d1841\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") "
Oct 31 03:43:01.638084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.637984    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-envoy\") pod \"d698f07e-e83e-4b52-9aba-1699cf8d1841\" (UID: \"d698f07e-e83e-4b52-9aba-1699cf8d1841\") "
Oct 31 03:43:01.638084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.637949    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-data" pod "d698f07e-e83e-4b52-9aba-1699cf8d1841" (UID: "d698f07e-e83e-4b52-9aba-1699cf8d1841"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:01.638084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.638034    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-workload-socket" pod "d698f07e-e83e-4b52-9aba-1699cf8d1841" (UID: "d698f07e-e83e-4b52-9aba-1699cf8d1841"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:01.638461 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.638204    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-workload-certs" pod "d698f07e-e83e-4b52-9aba-1699cf8d1841" (UID: "d698f07e-e83e-4b52-9aba-1699cf8d1841"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:01.638461 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.638220    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-credential-socket" pod "d698f07e-e83e-4b52-9aba-1699cf8d1841" (UID: "d698f07e-e83e-4b52-9aba-1699cf8d1841"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:01.638461 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.638307    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d698f07e-e83e-4b52-9aba-1699cf8d1841-istiod-ca-cert" pod "d698f07e-e83e-4b52-9aba-1699cf8d1841" (UID: "d698f07e-e83e-4b52-9aba-1699cf8d1841"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:01.638691 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.638660    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.638691 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.638688    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d698f07e-e83e-4b52-9aba-1699cf8d1841-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.638839 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.638705    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.638839 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.638719    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.638839 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.638734    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.640097 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.640065    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-podinfo" pod "d698f07e-e83e-4b52-9aba-1699cf8d1841" (UID: "d698f07e-e83e-4b52-9aba-1699cf8d1841"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:43:01.640299 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.640277    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-envoy" pod "d698f07e-e83e-4b52-9aba-1699cf8d1841" (UID: "d698f07e-e83e-4b52-9aba-1699cf8d1841"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:01.640299 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.640291    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-token" pod "d698f07e-e83e-4b52-9aba-1699cf8d1841" (UID: "d698f07e-e83e-4b52-9aba-1699cf8d1841"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:01.640411 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.640341    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d698f07e-e83e-4b52-9aba-1699cf8d1841-kube-api-access-7tdfk" pod "d698f07e-e83e-4b52-9aba-1699cf8d1841" (UID: "d698f07e-e83e-4b52-9aba-1699cf8d1841"). InnerVolumeSpecName "kube-api-access-7tdfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:01.739915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.739881    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.739915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.739909    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.739915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.739922    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tdfk\" (UniqueName: \"kubernetes.io/projected/d698f07e-e83e-4b52-9aba-1699cf8d1841-kube-api-access-7tdfk\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.740147 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.739933    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d698f07e-e83e-4b52-9aba-1699cf8d1841-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:01.816873 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.816750    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" event={"ID":"5f07fc2e-b24b-46c6-9571-39282e304a53","Type":"ContainerStarted","Data":"92be4990e1d9c5cd34a4858711be9fd8dac16d003606ce688212a0f98261e67a"}
Oct 31 03:43:01.816873 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.816838    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" event={"ID":"5f07fc2e-b24b-46c6-9571-39282e304a53","Type":"ContainerStarted","Data":"63e0ff25d0f09b3d6a0b2447871f1c68f875c4aca6911c992a00a567f4ce4f9a"}
Oct 31 03:43:01.818069 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.818044    4900 generic.go:356] "Generic (PLEG): container finished" podID="d698f07e-e83e-4b52-9aba-1699cf8d1841" containerID="d2e4a7aab6d453d49f946247b0551e87244d7fe796e2fe16545fa08e37980bf6" exitCode=0
Oct 31 03:43:01.818192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.818083    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"
Oct 31 03:43:01.818192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.818088    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f" event={"ID":"d698f07e-e83e-4b52-9aba-1699cf8d1841","Type":"ContainerDied","Data":"d2e4a7aab6d453d49f946247b0551e87244d7fe796e2fe16545fa08e37980bf6"}
Oct 31 03:43:01.818192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.818149    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f" event={"ID":"d698f07e-e83e-4b52-9aba-1699cf8d1841","Type":"ContainerDied","Data":"2096caacc4da0dd617c8b0ea660d24cc50df034d750705db34b3067b0df465f8"}
Oct 31 03:43:01.818192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.818182    4900 scope.go:122] "RemoveContainer" containerID="d2e4a7aab6d453d49f946247b0551e87244d7fe796e2fe16545fa08e37980bf6"
Oct 31 03:43:01.828333 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.828316    4900 scope.go:122] "RemoveContainer" containerID="d2e4a7aab6d453d49f946247b0551e87244d7fe796e2fe16545fa08e37980bf6"
Oct 31 03:43:01.828610 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:43:01.828592    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2e4a7aab6d453d49f946247b0551e87244d7fe796e2fe16545fa08e37980bf6\": container with ID starting with d2e4a7aab6d453d49f946247b0551e87244d7fe796e2fe16545fa08e37980bf6 not found: ID does not exist" containerID="d2e4a7aab6d453d49f946247b0551e87244d7fe796e2fe16545fa08e37980bf6"
Oct 31 03:43:01.828656 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.828620    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2e4a7aab6d453d49f946247b0551e87244d7fe796e2fe16545fa08e37980bf6"} err="failed to get container status \"d2e4a7aab6d453d49f946247b0551e87244d7fe796e2fe16545fa08e37980bf6\": rpc error: code = NotFound desc = could not find container \"d2e4a7aab6d453d49f946247b0551e87244d7fe796e2fe16545fa08e37980bf6\": container with ID starting with d2e4a7aab6d453d49f946247b0551e87244d7fe796e2fe16545fa08e37980bf6 not found: ID does not exist"
Oct 31 03:43:01.929579 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:01.929517    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" podStartSLOduration=1.929503932 podStartE2EDuration="1.929503932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:01.926487464 +0000 UTC m=+2134.892820820" watchObservedRunningTime="2026-10-31 03:43:01.929503932 +0000 UTC m=+2134.895837276"
Oct 31 03:43:02.031694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:02.031662    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"]
Oct 31 03:43:02.035053 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:02.035032    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-7j85f"]
Oct 31 03:43:02.592858 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:02.592829    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:02.599959 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:02.599930    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:02.825089 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:02.825044    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:02.826379 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:02.826355    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:43:03.152304 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:03.152269    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d698f07e-e83e-4b52-9aba-1699cf8d1841" path="/var/lib/kubelet/pods/d698f07e-e83e-4b52-9aba-1699cf8d1841/volumes"
Oct 31 03:43:03.448036 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:03.447924    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:05.050729 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:05.050691    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:05.389420 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:05.389313    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:06.984492 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:43:06.984457    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:43:07.937230 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:07.937189    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:08.474906 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:08.474866    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:08.678176 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:08.678135    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:08.999963 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:08.999934    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"]
Oct 31 03:43:09.000222 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.000199    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9" podUID="89ff4be5-188e-4c34-9627-fad9f2a7f296" containerName="maas-api" containerID="cri-o://dd5a50d19b99054927636b54c920f1d4b6259fb955c2cd681061b94ae91168f7" gracePeriod=30
Oct 31 03:43:09.024909 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.024878    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:09.244049 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.243488    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:09.283848 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.283821    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:43:09.347149 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.347109    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89ff4be5-188e-4c34-9627-fad9f2a7f296-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ff4be5-188e-4c34-9627-fad9f2a7f296-openshift-service-ca\") pod \"89ff4be5-188e-4c34-9627-fad9f2a7f296\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") "
Oct 31 03:43:09.347348 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.347232    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ff4be5-188e-4c34-9627-fad9f2a7f296-kube-api-access-6v98q\" (UniqueName: \"kubernetes.io/projected/89ff4be5-188e-4c34-9627-fad9f2a7f296-kube-api-access-6v98q\") pod \"89ff4be5-188e-4c34-9627-fad9f2a7f296\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") "
Oct 31 03:43:09.347348 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.347298    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-maas-api-tls\") pod \"89ff4be5-188e-4c34-9627-fad9f2a7f296\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") "
Oct 31 03:43:09.347348 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.347338    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-metrics-certs\") pod \"89ff4be5-188e-4c34-9627-fad9f2a7f296\" (UID: \"89ff4be5-188e-4c34-9627-fad9f2a7f296\") "
Oct 31 03:43:09.347504 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.347469    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89ff4be5-188e-4c34-9627-fad9f2a7f296-openshift-service-ca" pod "89ff4be5-188e-4c34-9627-fad9f2a7f296" (UID: "89ff4be5-188e-4c34-9627-fad9f2a7f296"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:09.347834 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.347813    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89ff4be5-188e-4c34-9627-fad9f2a7f296-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:09.349419 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.349394    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-maas-api-tls" pod "89ff4be5-188e-4c34-9627-fad9f2a7f296" (UID: "89ff4be5-188e-4c34-9627-fad9f2a7f296"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:43:09.349503 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.349423    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-metrics-certs" pod "89ff4be5-188e-4c34-9627-fad9f2a7f296" (UID: "89ff4be5-188e-4c34-9627-fad9f2a7f296"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:43:09.349503 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.349448    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ff4be5-188e-4c34-9627-fad9f2a7f296-kube-api-access-6v98q" pod "89ff4be5-188e-4c34-9627-fad9f2a7f296" (UID: "89ff4be5-188e-4c34-9627-fad9f2a7f296"). InnerVolumeSpecName "kube-api-access-6v98q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:09.448851 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.448807    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v98q\" (UniqueName: \"kubernetes.io/projected/89ff4be5-188e-4c34-9627-fad9f2a7f296-kube-api-access-6v98q\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:09.448851 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.448847    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:09.449070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.448863    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89ff4be5-188e-4c34-9627-fad9f2a7f296-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:09.874485 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.874442    4900 generic.go:356] "Generic (PLEG): container finished" podID="89ff4be5-188e-4c34-9627-fad9f2a7f296" containerID="dd5a50d19b99054927636b54c920f1d4b6259fb955c2cd681061b94ae91168f7" exitCode=0
Oct 31 03:43:09.874485 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.874490    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"
Oct 31 03:43:09.875054 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.874533    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9" event={"ID":"89ff4be5-188e-4c34-9627-fad9f2a7f296","Type":"ContainerDied","Data":"dd5a50d19b99054927636b54c920f1d4b6259fb955c2cd681061b94ae91168f7"}
Oct 31 03:43:09.875054 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.874601    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9" event={"ID":"89ff4be5-188e-4c34-9627-fad9f2a7f296","Type":"ContainerDied","Data":"35a1a23fc586e9ae0eccd2194def3c3a3f907b33ade77ad6cb9867e4c89be0b4"}
Oct 31 03:43:09.875054 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.874634    4900 scope.go:122] "RemoveContainer" containerID="dd5a50d19b99054927636b54c920f1d4b6259fb955c2cd681061b94ae91168f7"
Oct 31 03:43:09.884133 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.884114    4900 scope.go:122] "RemoveContainer" containerID="dd5a50d19b99054927636b54c920f1d4b6259fb955c2cd681061b94ae91168f7"
Oct 31 03:43:09.884412 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:43:09.884393    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5a50d19b99054927636b54c920f1d4b6259fb955c2cd681061b94ae91168f7\": container with ID starting with dd5a50d19b99054927636b54c920f1d4b6259fb955c2cd681061b94ae91168f7 not found: ID does not exist" containerID="dd5a50d19b99054927636b54c920f1d4b6259fb955c2cd681061b94ae91168f7"
Oct 31 03:43:09.884457 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:09.884422    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5a50d19b99054927636b54c920f1d4b6259fb955c2cd681061b94ae91168f7"} err="failed to get container status \"dd5a50d19b99054927636b54c920f1d4b6259fb955c2cd681061b94ae91168f7\": rpc error: code = NotFound desc = could not find container \"dd5a50d19b99054927636b54c920f1d4b6259fb955c2cd681061b94ae91168f7\": container with ID starting with dd5a50d19b99054927636b54c920f1d4b6259fb955c2cd681061b94ae91168f7 not found: ID does not exist"
Oct 31 03:43:10.028805 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:10.028761    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"]
Oct 31 03:43:10.074916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:10.074883    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-329d79-89d686b99-f9mv9"]
Oct 31 03:43:10.988177 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:10.988144    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:11.153145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:11.153110    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89ff4be5-188e-4c34-9627-fad9f2a7f296" path="/var/lib/kubelet/pods/89ff4be5-188e-4c34-9627-fad9f2a7f296/volumes"
Oct 31 03:43:11.233262 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:11.233231    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:11.453478 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:11.453378    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:12.365955 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:12.365407    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:12.583986 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:12.583958    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:12.797984 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:12.797954    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:12.940835 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:12.940808    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-k2qfn_0b2b721d-414f-449f-ba75-918fb3a59972/payload-processing/0.log"
Oct 31 03:43:13.011464 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:13.011432    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:13.202687 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:13.202604    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-k2qfn_0b2b721d-414f-449f-ba75-918fb3a59972/payload-processing/0.log"
Oct 31 03:43:13.967421 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:13.964530    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:14.512255 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:14.512213    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:14.978390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:14.978309    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:15.189248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:15.189216    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:15.409248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:15.409216    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:16.434017 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:16.433972    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:17.640433 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:17.640392    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:17.987172 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:17.987144    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:18.095010 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:18.094980    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"]
Oct 31 03:43:18.095225 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:18.095204    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs" podUID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" containerName="istio-proxy" containerID="cri-o://f4942df48ca71bca14c3f7e571c96166719bad5a39471cccd8eb18fb1bad1e9d" gracePeriod=30
Oct 31 03:43:18.201215 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:18.201174    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:19.832926 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:19.832891    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:20.467373 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:20.467320    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:20.670589 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:20.670555    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:21.031783 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:21.031744    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:21.864651 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:21.864616    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:22.081182 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:22.081147    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:22.495920 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:22.495882    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:22.746041 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:22.745963    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:22.979154 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:22.979127    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:23.371935 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.371819    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:43:23.397637 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.397607    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:23.452809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.452758    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-envoy\") pod \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") "
Oct 31 03:43:23.452978 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.452880    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-workload-certs\") pod \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") "
Oct 31 03:43:23.452978 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.452926    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istiod-ca-cert\") pod \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") "
Oct 31 03:43:23.453084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.452985    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-workload-socket\") pod \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") "
Oct 31 03:43:23.453084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453008    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-kube-api-access-mb5n6\" (UniqueName: \"kubernetes.io/projected/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-kube-api-access-mb5n6\") pod \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") "
Oct 31 03:43:23.453084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453040    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-data\") pod \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") "
Oct 31 03:43:23.453242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453120    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-token\" (UniqueName: \"kubernetes.io/projected/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-token\") pod \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") "
Oct 31 03:43:23.453242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453140    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-workload-certs" pod "0e56ee62-55fe-4c65-98d2-15ef41f72bb5" (UID: "0e56ee62-55fe-4c65-98d2-15ef41f72bb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:23.453242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453177    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-podinfo\") pod \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") "
Oct 31 03:43:23.453242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453208    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-credential-socket\") pod \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\" (UID: \"0e56ee62-55fe-4c65-98d2-15ef41f72bb5\") "
Oct 31 03:43:23.453454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453323    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-workload-socket" pod "0e56ee62-55fe-4c65-98d2-15ef41f72bb5" (UID: "0e56ee62-55fe-4c65-98d2-15ef41f72bb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:23.453454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453432    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istiod-ca-cert" pod "0e56ee62-55fe-4c65-98d2-15ef41f72bb5" (UID: "0e56ee62-55fe-4c65-98d2-15ef41f72bb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:23.453560 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453442    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-data" pod "0e56ee62-55fe-4c65-98d2-15ef41f72bb5" (UID: "0e56ee62-55fe-4c65-98d2-15ef41f72bb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:23.453655 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453633    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-credential-socket" pod "0e56ee62-55fe-4c65-98d2-15ef41f72bb5" (UID: "0e56ee62-55fe-4c65-98d2-15ef41f72bb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:23.453850 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453832    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:23.453908 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453851    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:23.453908 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453861    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:23.453908 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453871    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:23.453908 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.453884    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:23.455553 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.455530    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-envoy" pod "0e56ee62-55fe-4c65-98d2-15ef41f72bb5" (UID: "0e56ee62-55fe-4c65-98d2-15ef41f72bb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:23.455655 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.455538    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-token" pod "0e56ee62-55fe-4c65-98d2-15ef41f72bb5" (UID: "0e56ee62-55fe-4c65-98d2-15ef41f72bb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:23.455655 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.455625    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-podinfo" pod "0e56ee62-55fe-4c65-98d2-15ef41f72bb5" (UID: "0e56ee62-55fe-4c65-98d2-15ef41f72bb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:43:23.455655 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.455625    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-kube-api-access-mb5n6" pod "0e56ee62-55fe-4c65-98d2-15ef41f72bb5" (UID: "0e56ee62-55fe-4c65-98d2-15ef41f72bb5"). InnerVolumeSpecName "kube-api-access-mb5n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:23.558767 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.558730    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb5n6\" (UniqueName: \"kubernetes.io/projected/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-kube-api-access-mb5n6\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:23.558767 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.558767    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:23.559064 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.558781    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:23.559064 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.558812    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e56ee62-55fe-4c65-98d2-15ef41f72bb5-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:23.617300 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.617266    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:23.959448 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.959411    4900 generic.go:356] "Generic (PLEG): container finished" podID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" containerID="f4942df48ca71bca14c3f7e571c96166719bad5a39471cccd8eb18fb1bad1e9d" exitCode=0
Oct 31 03:43:23.959616 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.959461    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"
Oct 31 03:43:23.959616 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.959497    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs" event={"ID":"0e56ee62-55fe-4c65-98d2-15ef41f72bb5","Type":"ContainerDied","Data":"f4942df48ca71bca14c3f7e571c96166719bad5a39471cccd8eb18fb1bad1e9d"}
Oct 31 03:43:23.959616 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.959551    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs" event={"ID":"0e56ee62-55fe-4c65-98d2-15ef41f72bb5","Type":"ContainerDied","Data":"e2768800cb948feadc5d9c33e219b5d3ec460b85b125d331efd8cb8134d4f75a"}
Oct 31 03:43:23.959616 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.959576    4900 scope.go:122] "RemoveContainer" containerID="f4942df48ca71bca14c3f7e571c96166719bad5a39471cccd8eb18fb1bad1e9d"
Oct 31 03:43:23.969969 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.969943    4900 scope.go:122] "RemoveContainer" containerID="c1cc9234a64c15ae7fdb69bd6013b6720b0921391d19e5d5b39e085a8446b87f"
Oct 31 03:43:23.978482 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.978463    4900 scope.go:122] "RemoveContainer" containerID="f4942df48ca71bca14c3f7e571c96166719bad5a39471cccd8eb18fb1bad1e9d"
Oct 31 03:43:23.978771 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:43:23.978753    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4942df48ca71bca14c3f7e571c96166719bad5a39471cccd8eb18fb1bad1e9d\": container with ID starting with f4942df48ca71bca14c3f7e571c96166719bad5a39471cccd8eb18fb1bad1e9d not found: ID does not exist" containerID="f4942df48ca71bca14c3f7e571c96166719bad5a39471cccd8eb18fb1bad1e9d"
Oct 31 03:43:23.978838 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.978779    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4942df48ca71bca14c3f7e571c96166719bad5a39471cccd8eb18fb1bad1e9d"} err="failed to get container status \"f4942df48ca71bca14c3f7e571c96166719bad5a39471cccd8eb18fb1bad1e9d\": rpc error: code = NotFound desc = could not find container \"f4942df48ca71bca14c3f7e571c96166719bad5a39471cccd8eb18fb1bad1e9d\": container with ID starting with f4942df48ca71bca14c3f7e571c96166719bad5a39471cccd8eb18fb1bad1e9d not found: ID does not exist"
Oct 31 03:43:23.978838 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.978809    4900 scope.go:122] "RemoveContainer" containerID="c1cc9234a64c15ae7fdb69bd6013b6720b0921391d19e5d5b39e085a8446b87f"
Oct 31 03:43:23.979064 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:43:23.979045    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1cc9234a64c15ae7fdb69bd6013b6720b0921391d19e5d5b39e085a8446b87f\": container with ID starting with c1cc9234a64c15ae7fdb69bd6013b6720b0921391d19e5d5b39e085a8446b87f not found: ID does not exist" containerID="c1cc9234a64c15ae7fdb69bd6013b6720b0921391d19e5d5b39e085a8446b87f"
Oct 31 03:43:23.979105 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:23.979070    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1cc9234a64c15ae7fdb69bd6013b6720b0921391d19e5d5b39e085a8446b87f"} err="failed to get container status \"c1cc9234a64c15ae7fdb69bd6013b6720b0921391d19e5d5b39e085a8446b87f\": rpc error: code = NotFound desc = could not find container \"c1cc9234a64c15ae7fdb69bd6013b6720b0921391d19e5d5b39e085a8446b87f\": container with ID starting with c1cc9234a64c15ae7fdb69bd6013b6720b0921391d19e5d5b39e085a8446b87f not found: ID does not exist"
Oct 31 03:43:24.113085 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:24.113056    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"]
Oct 31 03:43:24.116040 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:24.116016    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c35264-openshift-default-6cf7474567-6wrhs"]
Oct 31 03:43:25.153708 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:25.153675    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" path="/var/lib/kubelet/pods/0e56ee62-55fe-4c65-98d2-15ef41f72bb5/volumes"
Oct 31 03:43:25.229294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:25.229251    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:25.527139 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:25.527095    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:26.809669 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:26.809634    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:27.328076 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:27.328045    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:28.604980 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:28.604941    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:28.830216 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:28.830187    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:29.124377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:29.124341    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:29.384360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:29.384282    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:29.645170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:29.645090    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:29.867703 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:29.867652    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:30.934931 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:30.934899    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:31.213698 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:31.213621    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:31.775811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:31.771976    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:32.801327 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:32.801291    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:33.422348 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:33.422298    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:33.704435 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:33.704353    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:34.022584 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:34.022548    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:34.317213 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:34.317082    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:36.827952 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:36.827885    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:36.984836 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:43:36.984767    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:43:38.254924 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:38.254889    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:38.536994 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:38.536905    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:41.608618 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:41.608540    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:41.812034 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:41.811997    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:42.296113 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:42.296077    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:47.230208 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:47.230173    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:52.134062 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:52.134016    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:53.596057 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:53.596016    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:54.446426 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:54.446389    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:43:57.164398 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:43:57.164367    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:00.772138 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:00.772107    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:01.292723 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:01.292685    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:02.289219 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.289183    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"]
Oct 31 03:44:02.289903 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.289883    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" containerName="istio-proxy"
Oct 31 03:44:02.290021 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.289904    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" containerName="istio-proxy"
Oct 31 03:44:02.290021 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.289960    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" containerName="istio-proxy"
Oct 31 03:44:02.290021 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.289968    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" containerName="istio-proxy"
Oct 31 03:44:02.290021 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.289983    4900 cpu_manager.go:414] "containerMap: removing container" podUID="d698f07e-e83e-4b52-9aba-1699cf8d1841" containerName="istio-proxy"
Oct 31 03:44:02.290021 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.289991    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d698f07e-e83e-4b52-9aba-1699cf8d1841" containerName="istio-proxy"
Oct 31 03:44:02.290021 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.290014    4900 cpu_manager.go:414] "containerMap: removing container" podUID="89ff4be5-188e-4c34-9627-fad9f2a7f296" containerName="maas-api"
Oct 31 03:44:02.290021 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.290022    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ff4be5-188e-4c34-9627-fad9f2a7f296" containerName="maas-api"
Oct 31 03:44:02.290368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.290137    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" containerName="istio-proxy"
Oct 31 03:44:02.290368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.290157    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e56ee62-55fe-4c65-98d2-15ef41f72bb5" containerName="istio-proxy"
Oct 31 03:44:02.290368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.290167    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="d698f07e-e83e-4b52-9aba-1699cf8d1841" containerName="istio-proxy"
Oct 31 03:44:02.290368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.290178    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ff4be5-188e-4c34-9627-fad9f2a7f296" containerName="maas-api"
Oct 31 03:44:02.293739 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.293719    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"
Oct 31 03:44:02.358534 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.358507    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72b34a0-8246-4c13-a3a3-49501c50d6e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2\" (UID: \"c72b34a0-8246-4c13-a3a3-49501c50d6e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"
Oct 31 03:44:02.358719 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.358695    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkpfx\" (UniqueName: \"kubernetes.io/projected/c72b34a0-8246-4c13-a3a3-49501c50d6e5-kube-api-access-rkpfx\") pod \"payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2\" (UID: \"c72b34a0-8246-4c13-a3a3-49501c50d6e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"
Oct 31 03:44:02.388796 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.388770    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c561e3\""
Oct 31 03:44:02.388940 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.388892    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c561e3-dockercfg-qwcq6\""
Oct 31 03:44:02.391372 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.391347    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"]
Oct 31 03:44:02.460110 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.460076    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72b34a0-8246-4c13-a3a3-49501c50d6e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2\" (UID: \"c72b34a0-8246-4c13-a3a3-49501c50d6e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"
Oct 31 03:44:02.460259 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.460225    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkpfx\" (UniqueName: \"kubernetes.io/projected/c72b34a0-8246-4c13-a3a3-49501c50d6e5-kube-api-access-rkpfx\") pod \"payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2\" (UID: \"c72b34a0-8246-4c13-a3a3-49501c50d6e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"
Oct 31 03:44:02.460843 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.460826    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72b34a0-8246-4c13-a3a3-49501c50d6e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2\" (UID: \"c72b34a0-8246-4c13-a3a3-49501c50d6e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"
Oct 31 03:44:02.513184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.513156    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkpfx\" (UniqueName: \"kubernetes.io/projected/c72b34a0-8246-4c13-a3a3-49501c50d6e5-kube-api-access-rkpfx\") pod \"payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2\" (UID: \"c72b34a0-8246-4c13-a3a3-49501c50d6e5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"
Oct 31 03:44:02.606555 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.606473    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"
Oct 31 03:44:02.970932 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:02.970909    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"]
Oct 31 03:44:02.972441 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:44:02.972377    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72b34a0_8246_4c13_a3a3_49501c50d6e5.slice/crio-933e09955772ce60fb2364b940eb6199841cfa64f81aa03a41f0f13246442641 WatchSource:0}: Error finding container 933e09955772ce60fb2364b940eb6199841cfa64f81aa03a41f0f13246442641: Status 404 returned error can't find the container with id 933e09955772ce60fb2364b940eb6199841cfa64f81aa03a41f0f13246442641
Oct 31 03:44:03.192869 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:03.192832    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2" event={"ID":"c72b34a0-8246-4c13-a3a3-49501c50d6e5","Type":"ContainerStarted","Data":"0dc5a751e02deddf8c003200ad5d989b637536e5ca41c0ebf72e481393f1e952"}
Oct 31 03:44:03.193065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:03.192892    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2" event={"ID":"c72b34a0-8246-4c13-a3a3-49501c50d6e5","Type":"ContainerStarted","Data":"933e09955772ce60fb2364b940eb6199841cfa64f81aa03a41f0f13246442641"}
Oct 31 03:44:03.193065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:03.193049    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"
Oct 31 03:44:03.303103 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:03.303040    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2" podStartSLOduration=1.303022012 podStartE2EDuration="1.303022012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:03.301350491 +0000 UTC m=+2196.267683836" watchObservedRunningTime="2026-10-31 03:44:03.303022012 +0000 UTC m=+2196.269355357"
Oct 31 03:44:03.814229 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:03.814185    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:04.027931 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:04.027892    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:04.999099 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:04.999066    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:05.206149 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:05.206106    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:06.985819 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:44:06.985758    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:44:08.872931 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:08.872895    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:13.713074 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:13.713039    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:14.200528 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:14.200497    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"
Oct 31 03:44:18.560498 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:18.560463    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:23.511526 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:23.511487    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:28.276111 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:28.276062    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:28.484175 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:28.484143    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:32.826510 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:32.826476    4900 patch_prober.go:29] interesting pod/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 03:44:32.826893 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:32.826531    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 03:44:33.357778 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:33.357746    4900 generic.go:356] "Generic (PLEG): container finished" podID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerID="92be4990e1d9c5cd34a4858711be9fd8dac16d003606ce688212a0f98261e67a" exitCode=0
Oct 31 03:44:33.357965 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:33.357808    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" event={"ID":"5f07fc2e-b24b-46c6-9571-39282e304a53","Type":"ContainerDied","Data":"92be4990e1d9c5cd34a4858711be9fd8dac16d003606ce688212a0f98261e67a"}
Oct 31 03:44:33.358199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:33.358185    4900 scope.go:122] "RemoveContainer" containerID="92be4990e1d9c5cd34a4858711be9fd8dac16d003606ce688212a0f98261e67a"
Oct 31 03:44:33.359650 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:33.359624    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:44:33.359713 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:33.359698    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:44:33.359749 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:33.359731    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:44:33.371072 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:33.371046    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:33.604512 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:33.604477    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:34.367577 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:34.367544    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" event={"ID":"5f07fc2e-b24b-46c6-9571-39282e304a53","Type":"ContainerStarted","Data":"3c1f59996c843325b3b3bf55df04542a14b8b39207cfe1f7f05fe9408923b137"}
Oct 31 03:44:34.593248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:34.593215    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:44:34.593529 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:34.593506    4900 patch_prober.go:29] interesting pod/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 03:44:34.593577 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:34.593554    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 03:44:35.593899 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:35.593861    4900 patch_prober.go:29] interesting pod/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 03:44:35.594313 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:35.593918    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 03:44:36.592971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:36.592936    4900 patch_prober.go:29] interesting pod/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 03:44:36.593150 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:36.593008    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 03:44:36.985536 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:44:36.985496    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:44:37.593402 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:37.593363    4900 patch_prober.go:29] interesting pod/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 03:44:37.593585 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:37.593431    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 03:44:38.593227 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:38.593181    4900 patch_prober.go:29] interesting pod/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 31 03:44:38.593764 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:38.593250    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 31 03:44:38.601902 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:38.601869    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:39.597833 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:39.597807    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:44:39.598278 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:39.598061    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:44:39.598657 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:39.598642    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:44:45.238417 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:45.238386    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:45.445664 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:45.445629    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:50.477346 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:50.477305    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:54.823968 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:54.823937    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:44:55.357191 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:44:55.357158    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:00.164542 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.164509    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"]
Oct 31 03:45:00.167412 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.167395    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"]
Oct 31 03:45:00.167532 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.167512    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"
Oct 31 03:45:00.170065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.170046    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"
Oct 31 03:45:00.184238 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.184217    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"]
Oct 31 03:45:00.186822 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.186777    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"
Oct 31 03:45:00.210650 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.210624    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"]
Oct 31 03:45:00.213286 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.213268    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"]
Oct 31 03:45:00.213391 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.213376    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"
Oct 31 03:45:00.215986 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.215966    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"
Oct 31 03:45:00.216767 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.216746    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"]
Oct 31 03:45:00.219897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.219881    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"
Oct 31 03:45:00.228135 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.228114    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 03:45:00.257248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.257228    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"]
Oct 31 03:45:00.259403 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.259385    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"]
Oct 31 03:45:00.271434 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.271410    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c47366-f8f3-4b1e-9f55-b48c7cd551a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk\" (UID: \"28c47366-f8f3-4b1e-9f55-b48c7cd551a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"
Oct 31 03:45:00.271535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.271477    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zj5w\" (UniqueName: \"kubernetes.io/projected/28c47366-f8f3-4b1e-9f55-b48c7cd551a3-kube-api-access-4zj5w\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk\" (UID: \"28c47366-f8f3-4b1e-9f55-b48c7cd551a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"
Oct 31 03:45:00.271535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.271498    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klz55\" (UniqueName: \"kubernetes.io/projected/214c92ee-f791-4685-bdee-19428df15788-kube-api-access-klz55\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9\" (UID: \"214c92ee-f791-4685-bdee-19428df15788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"
Oct 31 03:45:00.271860 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.271837    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214c92ee-f791-4685-bdee-19428df15788-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9\" (UID: \"214c92ee-f791-4685-bdee-19428df15788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"
Oct 31 03:45:00.315290 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.315257    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"]
Oct 31 03:45:00.318848 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.318818    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"]
Oct 31 03:45:00.358969 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.358935    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"]
Oct 31 03:45:00.360740 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.360711    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"]
Oct 31 03:45:00.373508 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.373468    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214c92ee-f791-4685-bdee-19428df15788-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9\" (UID: \"214c92ee-f791-4685-bdee-19428df15788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"
Oct 31 03:45:00.373736 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.373526    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx829\" (UniqueName: \"kubernetes.io/projected/2047069d-6fdb-4e59-8b0f-c181883e8bf0-kube-api-access-sx829\") pod \"maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb\" (UID: \"2047069d-6fdb-4e59-8b0f-c181883e8bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"
Oct 31 03:45:00.373736 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.373559    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ld9m\" (UniqueName: \"kubernetes.io/projected/c90fdf27-108a-436f-808d-c41495a6fb8c-kube-api-access-4ld9m\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8\" (UID: \"c90fdf27-108a-436f-808d-c41495a6fb8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"
Oct 31 03:45:00.373736 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.373589    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2047069d-6fdb-4e59-8b0f-c181883e8bf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb\" (UID: \"2047069d-6fdb-4e59-8b0f-c181883e8bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"
Oct 31 03:45:00.373736 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.373630    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c481e920-e2ce-47e4-8664-1f938fffca1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g\" (UID: \"c481e920-e2ce-47e4-8664-1f938fffca1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"
Oct 31 03:45:00.373736 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.373703    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c47366-f8f3-4b1e-9f55-b48c7cd551a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk\" (UID: \"28c47366-f8f3-4b1e-9f55-b48c7cd551a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"
Oct 31 03:45:00.373951 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.373751    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnh8f\" (UniqueName: \"kubernetes.io/projected/c481e920-e2ce-47e4-8664-1f938fffca1d-kube-api-access-fnh8f\") pod \"maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g\" (UID: \"c481e920-e2ce-47e4-8664-1f938fffca1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"
Oct 31 03:45:00.373951 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.373822    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a742c43-144f-4fad-bd47-8e4f74e4d129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85\" (UID: \"1a742c43-144f-4fad-bd47-8e4f74e4d129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"
Oct 31 03:45:00.373951 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.373874    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fctk\" (UniqueName: \"kubernetes.io/projected/1a742c43-144f-4fad-bd47-8e4f74e4d129-kube-api-access-5fctk\") pod \"maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85\" (UID: \"1a742c43-144f-4fad-bd47-8e4f74e4d129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"
Oct 31 03:45:00.373951 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.373907    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zj5w\" (UniqueName: \"kubernetes.io/projected/28c47366-f8f3-4b1e-9f55-b48c7cd551a3-kube-api-access-4zj5w\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk\" (UID: \"28c47366-f8f3-4b1e-9f55-b48c7cd551a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"
Oct 31 03:45:00.373951 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.373932    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klz55\" (UniqueName: \"kubernetes.io/projected/214c92ee-f791-4685-bdee-19428df15788-kube-api-access-klz55\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9\" (UID: \"214c92ee-f791-4685-bdee-19428df15788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"
Oct 31 03:45:00.374113 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.374031    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90fdf27-108a-436f-808d-c41495a6fb8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8\" (UID: \"c90fdf27-108a-436f-808d-c41495a6fb8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"
Oct 31 03:45:00.374946 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.374918    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214c92ee-f791-4685-bdee-19428df15788-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9\" (UID: \"214c92ee-f791-4685-bdee-19428df15788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"
Oct 31 03:45:00.375562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.375526    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c47366-f8f3-4b1e-9f55-b48c7cd551a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk\" (UID: \"28c47366-f8f3-4b1e-9f55-b48c7cd551a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"
Oct 31 03:45:00.386997 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.386970    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:00.429228 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.429164    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zj5w\" (UniqueName: \"kubernetes.io/projected/28c47366-f8f3-4b1e-9f55-b48c7cd551a3-kube-api-access-4zj5w\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk\" (UID: \"28c47366-f8f3-4b1e-9f55-b48c7cd551a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"
Oct 31 03:45:00.429334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.429237    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klz55\" (UniqueName: \"kubernetes.io/projected/214c92ee-f791-4685-bdee-19428df15788-kube-api-access-klz55\") pod \"maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9\" (UID: \"214c92ee-f791-4685-bdee-19428df15788\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"
Oct 31 03:45:00.475781 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.475754    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx829\" (UniqueName: \"kubernetes.io/projected/2047069d-6fdb-4e59-8b0f-c181883e8bf0-kube-api-access-sx829\") pod \"maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb\" (UID: \"2047069d-6fdb-4e59-8b0f-c181883e8bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"
Oct 31 03:45:00.475781 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.475798    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ld9m\" (UniqueName: \"kubernetes.io/projected/c90fdf27-108a-436f-808d-c41495a6fb8c-kube-api-access-4ld9m\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8\" (UID: \"c90fdf27-108a-436f-808d-c41495a6fb8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"
Oct 31 03:45:00.476000 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.475819    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2047069d-6fdb-4e59-8b0f-c181883e8bf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb\" (UID: \"2047069d-6fdb-4e59-8b0f-c181883e8bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"
Oct 31 03:45:00.476000 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.475844    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c481e920-e2ce-47e4-8664-1f938fffca1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g\" (UID: \"c481e920-e2ce-47e4-8664-1f938fffca1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"
Oct 31 03:45:00.476000 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.475898    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnh8f\" (UniqueName: \"kubernetes.io/projected/c481e920-e2ce-47e4-8664-1f938fffca1d-kube-api-access-fnh8f\") pod \"maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g\" (UID: \"c481e920-e2ce-47e4-8664-1f938fffca1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"
Oct 31 03:45:00.476000 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.475941    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a742c43-144f-4fad-bd47-8e4f74e4d129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85\" (UID: \"1a742c43-144f-4fad-bd47-8e4f74e4d129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"
Oct 31 03:45:00.476000 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.475989    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fctk\" (UniqueName: \"kubernetes.io/projected/1a742c43-144f-4fad-bd47-8e4f74e4d129-kube-api-access-5fctk\") pod \"maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85\" (UID: \"1a742c43-144f-4fad-bd47-8e4f74e4d129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"
Oct 31 03:45:00.476259 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.476082    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90fdf27-108a-436f-808d-c41495a6fb8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8\" (UID: \"c90fdf27-108a-436f-808d-c41495a6fb8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"
Oct 31 03:45:00.476642 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.476616    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a742c43-144f-4fad-bd47-8e4f74e4d129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85\" (UID: \"1a742c43-144f-4fad-bd47-8e4f74e4d129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"
Oct 31 03:45:00.476642 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.476634    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2047069d-6fdb-4e59-8b0f-c181883e8bf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb\" (UID: \"2047069d-6fdb-4e59-8b0f-c181883e8bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"
Oct 31 03:45:00.476752 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.476662    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c481e920-e2ce-47e4-8664-1f938fffca1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g\" (UID: \"c481e920-e2ce-47e4-8664-1f938fffca1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"
Oct 31 03:45:00.476815 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.476765    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90fdf27-108a-436f-808d-c41495a6fb8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8\" (UID: \"c90fdf27-108a-436f-808d-c41495a6fb8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"
Oct 31 03:45:00.480542 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.480527    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"
Oct 31 03:45:00.487075 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.487055    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"
Oct 31 03:45:00.530303 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.529911    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnh8f\" (UniqueName: \"kubernetes.io/projected/c481e920-e2ce-47e4-8664-1f938fffca1d-kube-api-access-fnh8f\") pod \"maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g\" (UID: \"c481e920-e2ce-47e4-8664-1f938fffca1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"
Oct 31 03:45:00.534583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.533661    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ld9m\" (UniqueName: \"kubernetes.io/projected/c90fdf27-108a-436f-808d-c41495a6fb8c-kube-api-access-4ld9m\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8\" (UID: \"c90fdf27-108a-436f-808d-c41495a6fb8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"
Oct 31 03:45:00.535369 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.535100    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fctk\" (UniqueName: \"kubernetes.io/projected/1a742c43-144f-4fad-bd47-8e4f74e4d129-kube-api-access-5fctk\") pod \"maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85\" (UID: \"1a742c43-144f-4fad-bd47-8e4f74e4d129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"
Oct 31 03:45:00.536825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.535527    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx829\" (UniqueName: \"kubernetes.io/projected/2047069d-6fdb-4e59-8b0f-c181883e8bf0-kube-api-access-sx829\") pod \"maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb\" (UID: \"2047069d-6fdb-4e59-8b0f-c181883e8bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"
Oct 31 03:45:00.555045 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.538034    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"
Oct 31 03:45:00.555045 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.543729    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"
Oct 31 03:45:00.798898 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.798867    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"
Oct 31 03:45:00.828543 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.828512    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"
Oct 31 03:45:00.846736 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.846712    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"]
Oct 31 03:45:00.848369 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:45:00.848340    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214c92ee_f791_4685_bdee_19428df15788.slice/crio-6c4ada48f05b27b5f7619e56e947ec7ffc1abab615ed418e1d2a16b857a895a5 WatchSource:0}: Error finding container 6c4ada48f05b27b5f7619e56e947ec7ffc1abab615ed418e1d2a16b857a895a5: Status 404 returned error can't find the container with id 6c4ada48f05b27b5f7619e56e947ec7ffc1abab615ed418e1d2a16b857a895a5
Oct 31 03:45:00.900108 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.900079    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"]
Oct 31 03:45:00.943070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.943041    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"]
Oct 31 03:45:00.946739 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:00.946721    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"]
Oct 31 03:45:00.946894 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:45:00.946870    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2047069d_6fdb_4e59_8b0f_c181883e8bf0.slice/crio-b1df3e4fb7d80540aa5266d1c2a52dd15d7f3a18a4cb91855bd98ade29f1b695 WatchSource:0}: Error finding container b1df3e4fb7d80540aa5266d1c2a52dd15d7f3a18a4cb91855bd98ade29f1b695: Status 404 returned error can't find the container with id b1df3e4fb7d80540aa5266d1c2a52dd15d7f3a18a4cb91855bd98ade29f1b695
Oct 31 03:45:01.169854 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.169826    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"]
Oct 31 03:45:01.171273 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:45:01.171250    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90fdf27_108a_436f_808d_c41495a6fb8c.slice/crio-2ebfd137c1c914ffab4c84b0766b643dc98d725dd75664c9d399cdf92fb9f0e0 WatchSource:0}: Error finding container 2ebfd137c1c914ffab4c84b0766b643dc98d725dd75664c9d399cdf92fb9f0e0: Status 404 returned error can't find the container with id 2ebfd137c1c914ffab4c84b0766b643dc98d725dd75664c9d399cdf92fb9f0e0
Oct 31 03:45:01.204927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.204905    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"]
Oct 31 03:45:01.205541 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:45:01.205518    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a742c43_144f_4fad_bd47_8e4f74e4d129.slice/crio-6de555a326fd15bd7ee7a7a6eb8ad7d609236d830cbf29d14c87270862c7e375 WatchSource:0}: Error finding container 6de555a326fd15bd7ee7a7a6eb8ad7d609236d830cbf29d14c87270862c7e375: Status 404 returned error can't find the container with id 6de555a326fd15bd7ee7a7a6eb8ad7d609236d830cbf29d14c87270862c7e375
Oct 31 03:45:01.534475 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.534438    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9" event={"ID":"214c92ee-f791-4685-bdee-19428df15788","Type":"ContainerStarted","Data":"de71e806c67aadc3e6beb5e2fe0390027416b3867e4c040c3fc7da1c55cfbed7"}
Oct 31 03:45:01.534747 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.534727    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9" event={"ID":"214c92ee-f791-4685-bdee-19428df15788","Type":"ContainerStarted","Data":"6c4ada48f05b27b5f7619e56e947ec7ffc1abab615ed418e1d2a16b857a895a5"}
Oct 31 03:45:01.536000 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.535968    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85" event={"ID":"1a742c43-144f-4fad-bd47-8e4f74e4d129","Type":"ContainerStarted","Data":"6de555a326fd15bd7ee7a7a6eb8ad7d609236d830cbf29d14c87270862c7e375"}
Oct 31 03:45:01.537698 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.537672    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g" event={"ID":"c481e920-e2ce-47e4-8664-1f938fffca1d","Type":"ContainerStarted","Data":"c7e8559f6e0ecfe1b02b8cf4132f387fdbf28fc4a3cfe770dcde1bf2cfd272f3"}
Oct 31 03:45:01.537828 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.537728    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g" event={"ID":"c481e920-e2ce-47e4-8664-1f938fffca1d","Type":"ContainerStarted","Data":"31804cedad171a73ce95d8f6aaf2c3a1f48fd2fe2c386ef704a30c7a6a5dd248"}
Oct 31 03:45:01.539370 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.539345    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb" event={"ID":"2047069d-6fdb-4e59-8b0f-c181883e8bf0","Type":"ContainerStarted","Data":"d49d08d82287b9cce98a300f7c3d861ce4305a7eb0b9ac8fa7002fd74e0e59b0"}
Oct 31 03:45:01.539474 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.539392    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb" event={"ID":"2047069d-6fdb-4e59-8b0f-c181883e8bf0","Type":"ContainerStarted","Data":"b1df3e4fb7d80540aa5266d1c2a52dd15d7f3a18a4cb91855bd98ade29f1b695"}
Oct 31 03:45:01.540918 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.540898    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8" event={"ID":"c90fdf27-108a-436f-808d-c41495a6fb8c","Type":"ContainerStarted","Data":"2ebfd137c1c914ffab4c84b0766b643dc98d725dd75664c9d399cdf92fb9f0e0"}
Oct 31 03:45:01.542472 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.542454    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk" event={"ID":"28c47366-f8f3-4b1e-9f55-b48c7cd551a3","Type":"ContainerStarted","Data":"8939ce6f06d4b4b358027f068cf9ec7284eec3f82c57c132fbc3818af3a07c3f"}
Oct 31 03:45:01.542542 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.542487    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk" event={"ID":"28c47366-f8f3-4b1e-9f55-b48c7cd551a3","Type":"ContainerStarted","Data":"6d4f06f50a7c18c857565554ff5de493d2b9717433eafec6a535afec7ace019a"}
Oct 31 03:45:01.667579 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.667522    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9" podStartSLOduration=1.6675072260000001 podStartE2EDuration="1.667507226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:01.664213977 +0000 UTC m=+2254.630547322" watchObservedRunningTime="2026-10-31 03:45:01.667507226 +0000 UTC m=+2254.633840569"
Oct 31 03:45:01.771455 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.771391    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk" podStartSLOduration=1.771371488 podStartE2EDuration="1.771371488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:01.770130738 +0000 UTC m=+2254.736464189" watchObservedRunningTime="2026-10-31 03:45:01.771371488 +0000 UTC m=+2254.737704835"
Oct 31 03:45:01.876914 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.876857    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb" podStartSLOduration=1.876840343 podStartE2EDuration="1.876840343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:01.875690192 +0000 UTC m=+2254.842023536" watchObservedRunningTime="2026-10-31 03:45:01.876840343 +0000 UTC m=+2254.843173688"
Oct 31 03:45:01.983477 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:01.983433    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g" podStartSLOduration=1.983417314 podStartE2EDuration="1.983417314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:01.982422103 +0000 UTC m=+2254.948755449" watchObservedRunningTime="2026-10-31 03:45:01.983417314 +0000 UTC m=+2254.949750693"
Oct 31 03:45:02.549963 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.549928    4900 generic.go:356] "Generic (PLEG): container finished" podID="214c92ee-f791-4685-bdee-19428df15788" containerID="de71e806c67aadc3e6beb5e2fe0390027416b3867e4c040c3fc7da1c55cfbed7" exitCode=60
Oct 31 03:45:02.550367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.550054    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9" event={"ID":"214c92ee-f791-4685-bdee-19428df15788","Type":"ContainerDied","Data":"de71e806c67aadc3e6beb5e2fe0390027416b3867e4c040c3fc7da1c55cfbed7"}
Oct 31 03:45:02.550367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.550300    4900 scope.go:122] "RemoveContainer" containerID="de71e806c67aadc3e6beb5e2fe0390027416b3867e4c040c3fc7da1c55cfbed7"
Oct 31 03:45:02.551635 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.551610    4900 generic.go:356] "Generic (PLEG): container finished" podID="1a742c43-144f-4fad-bd47-8e4f74e4d129" containerID="52feb86c5aeea9f7f4ef7ef8141def251beb047a728dc786096c68d487159cd6" exitCode=60
Oct 31 03:45:02.551739 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.551688    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85" event={"ID":"1a742c43-144f-4fad-bd47-8e4f74e4d129","Type":"ContainerDied","Data":"52feb86c5aeea9f7f4ef7ef8141def251beb047a728dc786096c68d487159cd6"}
Oct 31 03:45:02.551948 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.551930    4900 scope.go:122] "RemoveContainer" containerID="52feb86c5aeea9f7f4ef7ef8141def251beb047a728dc786096c68d487159cd6"
Oct 31 03:45:02.553346 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.553324    4900 generic.go:356] "Generic (PLEG): container finished" podID="c481e920-e2ce-47e4-8664-1f938fffca1d" containerID="c7e8559f6e0ecfe1b02b8cf4132f387fdbf28fc4a3cfe770dcde1bf2cfd272f3" exitCode=60
Oct 31 03:45:02.553418 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.553394    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g" event={"ID":"c481e920-e2ce-47e4-8664-1f938fffca1d","Type":"ContainerDied","Data":"c7e8559f6e0ecfe1b02b8cf4132f387fdbf28fc4a3cfe770dcde1bf2cfd272f3"}
Oct 31 03:45:02.558199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.553584    4900 scope.go:122] "RemoveContainer" containerID="c7e8559f6e0ecfe1b02b8cf4132f387fdbf28fc4a3cfe770dcde1bf2cfd272f3"
Oct 31 03:45:02.558199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.554978    4900 generic.go:356] "Generic (PLEG): container finished" podID="2047069d-6fdb-4e59-8b0f-c181883e8bf0" containerID="d49d08d82287b9cce98a300f7c3d861ce4305a7eb0b9ac8fa7002fd74e0e59b0" exitCode=60
Oct 31 03:45:02.558199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.555061    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb" event={"ID":"2047069d-6fdb-4e59-8b0f-c181883e8bf0","Type":"ContainerDied","Data":"d49d08d82287b9cce98a300f7c3d861ce4305a7eb0b9ac8fa7002fd74e0e59b0"}
Oct 31 03:45:02.558199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.555208    4900 scope.go:122] "RemoveContainer" containerID="d49d08d82287b9cce98a300f7c3d861ce4305a7eb0b9ac8fa7002fd74e0e59b0"
Oct 31 03:45:02.558199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.556442    4900 generic.go:356] "Generic (PLEG): container finished" podID="c90fdf27-108a-436f-808d-c41495a6fb8c" containerID="577b6856d7180615fb367cf0600e01eb21a226d3dc00f9858706d4699b668ca4" exitCode=60
Oct 31 03:45:02.558199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.556502    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8" event={"ID":"c90fdf27-108a-436f-808d-c41495a6fb8c","Type":"ContainerDied","Data":"577b6856d7180615fb367cf0600e01eb21a226d3dc00f9858706d4699b668ca4"}
Oct 31 03:45:02.558199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.556668    4900 scope.go:122] "RemoveContainer" containerID="577b6856d7180615fb367cf0600e01eb21a226d3dc00f9858706d4699b668ca4"
Oct 31 03:45:02.558199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.557768    4900 generic.go:356] "Generic (PLEG): container finished" podID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3" containerID="8939ce6f06d4b4b358027f068cf9ec7284eec3f82c57c132fbc3818af3a07c3f" exitCode=60
Oct 31 03:45:02.558199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.557810    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk" event={"ID":"28c47366-f8f3-4b1e-9f55-b48c7cd551a3","Type":"ContainerDied","Data":"8939ce6f06d4b4b358027f068cf9ec7284eec3f82c57c132fbc3818af3a07c3f"}
Oct 31 03:45:02.558199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:02.557984    4900 scope.go:122] "RemoveContainer" containerID="8939ce6f06d4b4b358027f068cf9ec7284eec3f82c57c132fbc3818af3a07c3f"
Oct 31 03:45:03.565096 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:03.565061    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85" event={"ID":"1a742c43-144f-4fad-bd47-8e4f74e4d129","Type":"ContainerStarted","Data":"9a16574712227b3296a37d25c26f18acf6abfa65834da1ba502eb883fe49e097"}
Oct 31 03:45:03.567008 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:03.566981    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g" event={"ID":"c481e920-e2ce-47e4-8664-1f938fffca1d","Type":"ContainerStarted","Data":"cc1061c7dc72eccf86ee8f37d69d58c49604a8910c7317dca4a2ac0d296e18af"}
Oct 31 03:45:03.573400 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:03.573377    4900 generic.go:356] "Generic (PLEG): container finished" podID="2047069d-6fdb-4e59-8b0f-c181883e8bf0" containerID="5d55872ef3711edfd9b300441d7dc7c378c579a57a399ccb4c86efb2d31b8b09" exitCode=60
Oct 31 03:45:03.573522 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:03.573456    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb" event={"ID":"2047069d-6fdb-4e59-8b0f-c181883e8bf0","Type":"ContainerDied","Data":"5d55872ef3711edfd9b300441d7dc7c378c579a57a399ccb4c86efb2d31b8b09"}
Oct 31 03:45:03.573522 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:03.573508    4900 scope.go:122] "RemoveContainer" containerID="d49d08d82287b9cce98a300f7c3d861ce4305a7eb0b9ac8fa7002fd74e0e59b0"
Oct 31 03:45:03.573760 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:03.573741    4900 scope.go:122] "RemoveContainer" containerID="5d55872ef3711edfd9b300441d7dc7c378c579a57a399ccb4c86efb2d31b8b09"
Oct 31 03:45:03.574590 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:03.574564    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb_odh-ai-gateway-infra(2047069d-6fdb-4e59-8b0f-c181883e8bf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb" podUID="2047069d-6fdb-4e59-8b0f-c181883e8bf0"
Oct 31 03:45:03.575780 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:03.575755    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8" event={"ID":"c90fdf27-108a-436f-808d-c41495a6fb8c","Type":"ContainerStarted","Data":"41ac00e5e10bd174bd291f5d20c2e216f8494b43a0230f6c7da46b219f2ad057"}
Oct 31 03:45:03.577844 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:03.577761    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk" event={"ID":"28c47366-f8f3-4b1e-9f55-b48c7cd551a3","Type":"ContainerStarted","Data":"b1472fec89759ee49fc6443112da8d5e50bc71250f0193113ccd932a624fd818"}
Oct 31 03:45:03.579412 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:03.579393    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9" event={"ID":"214c92ee-f791-4685-bdee-19428df15788","Type":"ContainerStarted","Data":"a8bffd65fe39c4faf4c6840a9d601aa0ee229d59ae599f543b7bf0715e44ce94"}
Oct 31 03:45:03.672042 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:03.671964    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85" podStartSLOduration=3.671946991 podStartE2EDuration="3.671946991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:03.671409484 +0000 UTC m=+2256.637742827" watchObservedRunningTime="2026-10-31 03:45:03.671946991 +0000 UTC m=+2256.638280334"
Oct 31 03:45:04.200596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.200500    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8" podStartSLOduration=4.200486287 podStartE2EDuration="4.200486287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:04.198651251 +0000 UTC m=+2257.164984599" watchObservedRunningTime="2026-10-31 03:45:04.200486287 +0000 UTC m=+2257.166819631"
Oct 31 03:45:04.586623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.586575    4900 generic.go:356] "Generic (PLEG): container finished" podID="c90fdf27-108a-436f-808d-c41495a6fb8c" containerID="41ac00e5e10bd174bd291f5d20c2e216f8494b43a0230f6c7da46b219f2ad057" exitCode=60
Oct 31 03:45:04.587088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.586676    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8" event={"ID":"c90fdf27-108a-436f-808d-c41495a6fb8c","Type":"ContainerDied","Data":"41ac00e5e10bd174bd291f5d20c2e216f8494b43a0230f6c7da46b219f2ad057"}
Oct 31 03:45:04.587088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.586767    4900 scope.go:122] "RemoveContainer" containerID="577b6856d7180615fb367cf0600e01eb21a226d3dc00f9858706d4699b668ca4"
Oct 31 03:45:04.587088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.586969    4900 scope.go:122] "RemoveContainer" containerID="41ac00e5e10bd174bd291f5d20c2e216f8494b43a0230f6c7da46b219f2ad057"
Oct 31 03:45:04.587275 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:04.587259    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8_odh-ai-gateway-infra(c90fdf27-108a-436f-808d-c41495a6fb8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8" podUID="c90fdf27-108a-436f-808d-c41495a6fb8c"
Oct 31 03:45:04.588614 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.588587    4900 generic.go:356] "Generic (PLEG): container finished" podID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3" containerID="b1472fec89759ee49fc6443112da8d5e50bc71250f0193113ccd932a624fd818" exitCode=60
Oct 31 03:45:04.588750 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.588658    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk" event={"ID":"28c47366-f8f3-4b1e-9f55-b48c7cd551a3","Type":"ContainerDied","Data":"b1472fec89759ee49fc6443112da8d5e50bc71250f0193113ccd932a624fd818"}
Oct 31 03:45:04.588877 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.588862    4900 scope.go:122] "RemoveContainer" containerID="b1472fec89759ee49fc6443112da8d5e50bc71250f0193113ccd932a624fd818"
Oct 31 03:45:04.589112 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:04.589092    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk_odh-ai-gateway-infra(28c47366-f8f3-4b1e-9f55-b48c7cd551a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk" podUID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3"
Oct 31 03:45:04.590632 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.590611    4900 generic.go:356] "Generic (PLEG): container finished" podID="214c92ee-f791-4685-bdee-19428df15788" containerID="a8bffd65fe39c4faf4c6840a9d601aa0ee229d59ae599f543b7bf0715e44ce94" exitCode=60
Oct 31 03:45:04.590719 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.590694    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9" event={"ID":"214c92ee-f791-4685-bdee-19428df15788","Type":"ContainerDied","Data":"a8bffd65fe39c4faf4c6840a9d601aa0ee229d59ae599f543b7bf0715e44ce94"}
Oct 31 03:45:04.590962 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.590939    4900 scope.go:122] "RemoveContainer" containerID="a8bffd65fe39c4faf4c6840a9d601aa0ee229d59ae599f543b7bf0715e44ce94"
Oct 31 03:45:04.591252 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:04.591230    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9_odh-ai-gateway-infra(214c92ee-f791-4685-bdee-19428df15788)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9" podUID="214c92ee-f791-4685-bdee-19428df15788"
Oct 31 03:45:04.592878 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.592859    4900 generic.go:356] "Generic (PLEG): container finished" podID="1a742c43-144f-4fad-bd47-8e4f74e4d129" containerID="9a16574712227b3296a37d25c26f18acf6abfa65834da1ba502eb883fe49e097" exitCode=60
Oct 31 03:45:04.592973 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.592887    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85" event={"ID":"1a742c43-144f-4fad-bd47-8e4f74e4d129","Type":"ContainerDied","Data":"9a16574712227b3296a37d25c26f18acf6abfa65834da1ba502eb883fe49e097"}
Oct 31 03:45:04.593152 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.593132    4900 scope.go:122] "RemoveContainer" containerID="9a16574712227b3296a37d25c26f18acf6abfa65834da1ba502eb883fe49e097"
Oct 31 03:45:04.593406 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:04.593384    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85_odh-ai-gateway-infra(1a742c43-144f-4fad-bd47-8e4f74e4d129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85" podUID="1a742c43-144f-4fad-bd47-8e4f74e4d129"
Oct 31 03:45:04.594928 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.594906    4900 generic.go:356] "Generic (PLEG): container finished" podID="c481e920-e2ce-47e4-8664-1f938fffca1d" containerID="cc1061c7dc72eccf86ee8f37d69d58c49604a8910c7317dca4a2ac0d296e18af" exitCode=60
Oct 31 03:45:04.594998 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.594968    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g" event={"ID":"c481e920-e2ce-47e4-8664-1f938fffca1d","Type":"ContainerDied","Data":"cc1061c7dc72eccf86ee8f37d69d58c49604a8910c7317dca4a2ac0d296e18af"}
Oct 31 03:45:04.595159 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.595144    4900 scope.go:122] "RemoveContainer" containerID="cc1061c7dc72eccf86ee8f37d69d58c49604a8910c7317dca4a2ac0d296e18af"
Oct 31 03:45:04.595370 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:04.595353    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g_odh-ai-gateway-infra(c481e920-e2ce-47e4-8664-1f938fffca1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g" podUID="c481e920-e2ce-47e4-8664-1f938fffca1d"
Oct 31 03:45:04.596729 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.596709    4900 scope.go:122] "RemoveContainer" containerID="5d55872ef3711edfd9b300441d7dc7c378c579a57a399ccb4c86efb2d31b8b09"
Oct 31 03:45:04.596913 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:04.596896    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb_odh-ai-gateway-infra(2047069d-6fdb-4e59-8b0f-c181883e8bf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb" podUID="2047069d-6fdb-4e59-8b0f-c181883e8bf0"
Oct 31 03:45:04.599116 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.599099    4900 scope.go:122] "RemoveContainer" containerID="8939ce6f06d4b4b358027f068cf9ec7284eec3f82c57c132fbc3818af3a07c3f"
Oct 31 03:45:04.609393 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.609375    4900 scope.go:122] "RemoveContainer" containerID="de71e806c67aadc3e6beb5e2fe0390027416b3867e4c040c3fc7da1c55cfbed7"
Oct 31 03:45:04.619442 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.619422    4900 scope.go:122] "RemoveContainer" containerID="52feb86c5aeea9f7f4ef7ef8141def251beb047a728dc786096c68d487159cd6"
Oct 31 03:45:04.629450 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:04.629430    4900 scope.go:122] "RemoveContainer" containerID="c7e8559f6e0ecfe1b02b8cf4132f387fdbf28fc4a3cfe770dcde1bf2cfd272f3"
Oct 31 03:45:05.353321 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:05.353287    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:05.605330 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:05.605252    4900 scope.go:122] "RemoveContainer" containerID="a8bffd65fe39c4faf4c6840a9d601aa0ee229d59ae599f543b7bf0715e44ce94"
Oct 31 03:45:05.605748 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:05.605563    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9_odh-ai-gateway-infra(214c92ee-f791-4685-bdee-19428df15788)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9" podUID="214c92ee-f791-4685-bdee-19428df15788"
Oct 31 03:45:05.606831 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:05.606781    4900 scope.go:122] "RemoveContainer" containerID="9a16574712227b3296a37d25c26f18acf6abfa65834da1ba502eb883fe49e097"
Oct 31 03:45:05.607069 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:05.607047    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85_odh-ai-gateway-infra(1a742c43-144f-4fad-bd47-8e4f74e4d129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85" podUID="1a742c43-144f-4fad-bd47-8e4f74e4d129"
Oct 31 03:45:05.608235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:05.608216    4900 scope.go:122] "RemoveContainer" containerID="cc1061c7dc72eccf86ee8f37d69d58c49604a8910c7317dca4a2ac0d296e18af"
Oct 31 03:45:05.608429 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:05.608411    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g_odh-ai-gateway-infra(c481e920-e2ce-47e4-8664-1f938fffca1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g" podUID="c481e920-e2ce-47e4-8664-1f938fffca1d"
Oct 31 03:45:05.609579 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:05.609560    4900 scope.go:122] "RemoveContainer" containerID="41ac00e5e10bd174bd291f5d20c2e216f8494b43a0230f6c7da46b219f2ad057"
Oct 31 03:45:05.609776 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:05.609758    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8_odh-ai-gateway-infra(c90fdf27-108a-436f-808d-c41495a6fb8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8" podUID="c90fdf27-108a-436f-808d-c41495a6fb8c"
Oct 31 03:45:05.610966 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:05.610951    4900 scope.go:122] "RemoveContainer" containerID="b1472fec89759ee49fc6443112da8d5e50bc71250f0193113ccd932a624fd818"
Oct 31 03:45:05.611128 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:05.611113    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk_odh-ai-gateway-infra(28c47366-f8f3-4b1e-9f55-b48c7cd551a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk" podUID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3"
Oct 31 03:45:06.985286 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:06.985250    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:45:10.422231 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:10.422142    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:15.153297 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:15.153255    4900 scope.go:122] "RemoveContainer" containerID="a8bffd65fe39c4faf4c6840a9d601aa0ee229d59ae599f543b7bf0715e44ce94"
Oct 31 03:45:15.153782 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:15.153415    4900 scope.go:122] "RemoveContainer" containerID="9a16574712227b3296a37d25c26f18acf6abfa65834da1ba502eb883fe49e097"
Oct 31 03:45:15.153782 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:15.153532    4900 scope.go:122] "RemoveContainer" containerID="5d55872ef3711edfd9b300441d7dc7c378c579a57a399ccb4c86efb2d31b8b09"
Oct 31 03:45:15.679751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:15.679711    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9" event={"ID":"214c92ee-f791-4685-bdee-19428df15788","Type":"ContainerStarted","Data":"d02ff9521085bfa03ec305f49b095535690acd6a9b302e9429688935ea78c5c1"}
Oct 31 03:45:15.681811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:15.681772    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85" event={"ID":"1a742c43-144f-4fad-bd47-8e4f74e4d129","Type":"ContainerStarted","Data":"970f325228fdeb26cbd46e0dd78abc3d8747a6b48e2a54120c22e1b7c97d00bc"}
Oct 31 03:45:15.683686 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:15.683664    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb" event={"ID":"2047069d-6fdb-4e59-8b0f-c181883e8bf0","Type":"ContainerStarted","Data":"789558946e6fb5812496367fc2dab5a59ffdb67a2ab20e49e58018e06ef332a6"}
Oct 31 03:45:16.066805 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.066764    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:16.315163 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.315133    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:16.691022 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.690947    4900 generic.go:356] "Generic (PLEG): container finished" podID="2047069d-6fdb-4e59-8b0f-c181883e8bf0" containerID="789558946e6fb5812496367fc2dab5a59ffdb67a2ab20e49e58018e06ef332a6" exitCode=60
Oct 31 03:45:16.691167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.691023    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb" event={"ID":"2047069d-6fdb-4e59-8b0f-c181883e8bf0","Type":"ContainerDied","Data":"789558946e6fb5812496367fc2dab5a59ffdb67a2ab20e49e58018e06ef332a6"}
Oct 31 03:45:16.691167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.691085    4900 scope.go:122] "RemoveContainer" containerID="5d55872ef3711edfd9b300441d7dc7c378c579a57a399ccb4c86efb2d31b8b09"
Oct 31 03:45:16.691414 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.691398    4900 scope.go:122] "RemoveContainer" containerID="789558946e6fb5812496367fc2dab5a59ffdb67a2ab20e49e58018e06ef332a6"
Oct 31 03:45:16.691660 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:16.691636    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb_odh-ai-gateway-infra(2047069d-6fdb-4e59-8b0f-c181883e8bf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb" podUID="2047069d-6fdb-4e59-8b0f-c181883e8bf0"
Oct 31 03:45:16.693153 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.693135    4900 generic.go:356] "Generic (PLEG): container finished" podID="214c92ee-f791-4685-bdee-19428df15788" containerID="d02ff9521085bfa03ec305f49b095535690acd6a9b302e9429688935ea78c5c1" exitCode=60
Oct 31 03:45:16.693244 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.693201    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9" event={"ID":"214c92ee-f791-4685-bdee-19428df15788","Type":"ContainerDied","Data":"d02ff9521085bfa03ec305f49b095535690acd6a9b302e9429688935ea78c5c1"}
Oct 31 03:45:16.693507 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.693481    4900 scope.go:122] "RemoveContainer" containerID="d02ff9521085bfa03ec305f49b095535690acd6a9b302e9429688935ea78c5c1"
Oct 31 03:45:16.693725 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:16.693703    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9_odh-ai-gateway-infra(214c92ee-f791-4685-bdee-19428df15788)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9" podUID="214c92ee-f791-4685-bdee-19428df15788"
Oct 31 03:45:16.695100 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.695080    4900 generic.go:356] "Generic (PLEG): container finished" podID="1a742c43-144f-4fad-bd47-8e4f74e4d129" containerID="970f325228fdeb26cbd46e0dd78abc3d8747a6b48e2a54120c22e1b7c97d00bc" exitCode=60
Oct 31 03:45:16.695204 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.695105    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85" event={"ID":"1a742c43-144f-4fad-bd47-8e4f74e4d129","Type":"ContainerDied","Data":"970f325228fdeb26cbd46e0dd78abc3d8747a6b48e2a54120c22e1b7c97d00bc"}
Oct 31 03:45:16.695368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.695354    4900 scope.go:122] "RemoveContainer" containerID="970f325228fdeb26cbd46e0dd78abc3d8747a6b48e2a54120c22e1b7c97d00bc"
Oct 31 03:45:16.695545 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:16.695527    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85_odh-ai-gateway-infra(1a742c43-144f-4fad-bd47-8e4f74e4d129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85" podUID="1a742c43-144f-4fad-bd47-8e4f74e4d129"
Oct 31 03:45:16.706469 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.706450    4900 scope.go:122] "RemoveContainer" containerID="a8bffd65fe39c4faf4c6840a9d601aa0ee229d59ae599f543b7bf0715e44ce94"
Oct 31 03:45:16.717148 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.717132    4900 scope.go:122] "RemoveContainer" containerID="9a16574712227b3296a37d25c26f18acf6abfa65834da1ba502eb883fe49e097"
Oct 31 03:45:16.796210 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.796173    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"]
Oct 31 03:45:16.901435 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:16.901398    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"]
Oct 31 03:45:17.007676 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:17.007645    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"]
Oct 31 03:45:17.154377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:17.154348    4900 scope.go:122] "RemoveContainer" containerID="b1472fec89759ee49fc6443112da8d5e50bc71250f0193113ccd932a624fd818"
Oct 31 03:45:17.154548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:17.154454    4900 scope.go:122] "RemoveContainer" containerID="41ac00e5e10bd174bd291f5d20c2e216f8494b43a0230f6c7da46b219f2ad057"
Oct 31 03:45:17.266108 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:17.266005    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:17.710772 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:17.710741    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8" event={"ID":"c90fdf27-108a-436f-808d-c41495a6fb8c","Type":"ContainerStarted","Data":"4284ef7d279427b0d1c12a6cc1900e26e5b3febb2876acaaf97f0163ad9f2123"}
Oct 31 03:45:17.713234 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:17.713205    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk" event={"ID":"28c47366-f8f3-4b1e-9f55-b48c7cd551a3","Type":"ContainerStarted","Data":"47a2df5fe66cfd36d2b4864a4c694cbc1a9c44d8b5f7a13d15d466d679fe20f1"}
Oct 31 03:45:17.871056 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:17.871023    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"
Oct 31 03:45:17.894251 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:17.894225    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"
Oct 31 03:45:17.924198 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:17.924175    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"
Oct 31 03:45:18.005848 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.005816    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2047069d-6fdb-4e59-8b0f-c181883e8bf0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2047069d-6fdb-4e59-8b0f-c181883e8bf0-maas-api-service-ca\") pod \"2047069d-6fdb-4e59-8b0f-c181883e8bf0\" (UID: \"2047069d-6fdb-4e59-8b0f-c181883e8bf0\") "
Oct 31 03:45:18.005989 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.005940    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/214c92ee-f791-4685-bdee-19428df15788-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214c92ee-f791-4685-bdee-19428df15788-maas-api-service-ca\") pod \"214c92ee-f791-4685-bdee-19428df15788\" (UID: \"214c92ee-f791-4685-bdee-19428df15788\") "
Oct 31 03:45:18.005989 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.005984    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/214c92ee-f791-4685-bdee-19428df15788-kube-api-access-klz55\" (UniqueName: \"kubernetes.io/projected/214c92ee-f791-4685-bdee-19428df15788-kube-api-access-klz55\") pod \"214c92ee-f791-4685-bdee-19428df15788\" (UID: \"214c92ee-f791-4685-bdee-19428df15788\") "
Oct 31 03:45:18.006114 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.006034    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2047069d-6fdb-4e59-8b0f-c181883e8bf0-kube-api-access-sx829\" (UniqueName: \"kubernetes.io/projected/2047069d-6fdb-4e59-8b0f-c181883e8bf0-kube-api-access-sx829\") pod \"2047069d-6fdb-4e59-8b0f-c181883e8bf0\" (UID: \"2047069d-6fdb-4e59-8b0f-c181883e8bf0\") "
Oct 31 03:45:18.006114 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.006058    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a742c43-144f-4fad-bd47-8e4f74e4d129-kube-api-access-5fctk\" (UniqueName: \"kubernetes.io/projected/1a742c43-144f-4fad-bd47-8e4f74e4d129-kube-api-access-5fctk\") pod \"1a742c43-144f-4fad-bd47-8e4f74e4d129\" (UID: \"1a742c43-144f-4fad-bd47-8e4f74e4d129\") "
Oct 31 03:45:18.006114 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.006086    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a742c43-144f-4fad-bd47-8e4f74e4d129-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a742c43-144f-4fad-bd47-8e4f74e4d129-maas-api-service-ca\") pod \"1a742c43-144f-4fad-bd47-8e4f74e4d129\" (UID: \"1a742c43-144f-4fad-bd47-8e4f74e4d129\") "
Oct 31 03:45:18.006289 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.006253    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2047069d-6fdb-4e59-8b0f-c181883e8bf0-maas-api-service-ca" pod "2047069d-6fdb-4e59-8b0f-c181883e8bf0" (UID: "2047069d-6fdb-4e59-8b0f-c181883e8bf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:18.006477 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.006370    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/214c92ee-f791-4685-bdee-19428df15788-maas-api-service-ca" pod "214c92ee-f791-4685-bdee-19428df15788" (UID: "214c92ee-f791-4685-bdee-19428df15788"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:18.006579 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.006522    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a742c43-144f-4fad-bd47-8e4f74e4d129-maas-api-service-ca" pod "1a742c43-144f-4fad-bd47-8e4f74e4d129" (UID: "1a742c43-144f-4fad-bd47-8e4f74e4d129"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:18.006667 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.006647    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a742c43-144f-4fad-bd47-8e4f74e4d129-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:18.006723 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.006670    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2047069d-6fdb-4e59-8b0f-c181883e8bf0-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:18.006723 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.006683    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214c92ee-f791-4685-bdee-19428df15788-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:18.008091 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.008071    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a742c43-144f-4fad-bd47-8e4f74e4d129-kube-api-access-5fctk" pod "1a742c43-144f-4fad-bd47-8e4f74e4d129" (UID: "1a742c43-144f-4fad-bd47-8e4f74e4d129"). InnerVolumeSpecName "kube-api-access-5fctk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:18.008525 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.008507    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2047069d-6fdb-4e59-8b0f-c181883e8bf0-kube-api-access-sx829" pod "2047069d-6fdb-4e59-8b0f-c181883e8bf0" (UID: "2047069d-6fdb-4e59-8b0f-c181883e8bf0"). InnerVolumeSpecName "kube-api-access-sx829". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:18.008579 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.008515    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214c92ee-f791-4685-bdee-19428df15788-kube-api-access-klz55" pod "214c92ee-f791-4685-bdee-19428df15788" (UID: "214c92ee-f791-4685-bdee-19428df15788"). InnerVolumeSpecName "kube-api-access-klz55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:18.107669 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.107576    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klz55\" (UniqueName: \"kubernetes.io/projected/214c92ee-f791-4685-bdee-19428df15788-kube-api-access-klz55\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:18.107669 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.107613    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx829\" (UniqueName: \"kubernetes.io/projected/2047069d-6fdb-4e59-8b0f-c181883e8bf0-kube-api-access-sx829\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:18.107669 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.107624    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fctk\" (UniqueName: \"kubernetes.io/projected/1a742c43-144f-4fad-bd47-8e4f74e4d129-kube-api-access-5fctk\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:18.146146 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.146125    4900 scope.go:122] "RemoveContainer" containerID="cc1061c7dc72eccf86ee8f37d69d58c49604a8910c7317dca4a2ac0d296e18af"
Oct 31 03:45:18.723595 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.723496    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g" event={"ID":"c481e920-e2ce-47e4-8664-1f938fffca1d","Type":"ContainerStarted","Data":"be2441309ca660689830d859ae9a5e93a038d8f760745766f2fcf12e10c85e42"}
Oct 31 03:45:18.724912 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.724893    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"
Oct 31 03:45:18.725023 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.724913    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb" event={"ID":"2047069d-6fdb-4e59-8b0f-c181883e8bf0","Type":"ContainerDied","Data":"b1df3e4fb7d80540aa5266d1c2a52dd15d7f3a18a4cb91855bd98ade29f1b695"}
Oct 31 03:45:18.725023 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.724962    4900 scope.go:122] "RemoveContainer" containerID="789558946e6fb5812496367fc2dab5a59ffdb67a2ab20e49e58018e06ef332a6"
Oct 31 03:45:18.727016 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.726989    4900 generic.go:356] "Generic (PLEG): container finished" podID="c90fdf27-108a-436f-808d-c41495a6fb8c" containerID="4284ef7d279427b0d1c12a6cc1900e26e5b3febb2876acaaf97f0163ad9f2123" exitCode=60
Oct 31 03:45:18.727114 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.727051    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8" event={"ID":"c90fdf27-108a-436f-808d-c41495a6fb8c","Type":"ContainerDied","Data":"4284ef7d279427b0d1c12a6cc1900e26e5b3febb2876acaaf97f0163ad9f2123"}
Oct 31 03:45:18.727455 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.727436    4900 scope.go:122] "RemoveContainer" containerID="4284ef7d279427b0d1c12a6cc1900e26e5b3febb2876acaaf97f0163ad9f2123"
Oct 31 03:45:18.727745 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:18.727721    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8_odh-ai-gateway-infra(c90fdf27-108a-436f-808d-c41495a6fb8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8" podUID="c90fdf27-108a-436f-808d-c41495a6fb8c"
Oct 31 03:45:18.729101 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.729082    4900 generic.go:356] "Generic (PLEG): container finished" podID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3" containerID="47a2df5fe66cfd36d2b4864a4c694cbc1a9c44d8b5f7a13d15d466d679fe20f1" exitCode=60
Oct 31 03:45:18.729182 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.729158    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk" event={"ID":"28c47366-f8f3-4b1e-9f55-b48c7cd551a3","Type":"ContainerDied","Data":"47a2df5fe66cfd36d2b4864a4c694cbc1a9c44d8b5f7a13d15d466d679fe20f1"}
Oct 31 03:45:18.729479 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.729461    4900 scope.go:122] "RemoveContainer" containerID="47a2df5fe66cfd36d2b4864a4c694cbc1a9c44d8b5f7a13d15d466d679fe20f1"
Oct 31 03:45:18.729709 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:18.729688    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk_odh-ai-gateway-infra(28c47366-f8f3-4b1e-9f55-b48c7cd551a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk" podUID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3"
Oct 31 03:45:18.730654 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.730636    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"
Oct 31 03:45:18.730737 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.730666    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9" event={"ID":"214c92ee-f791-4685-bdee-19428df15788","Type":"ContainerDied","Data":"6c4ada48f05b27b5f7619e56e947ec7ffc1abab615ed418e1d2a16b857a895a5"}
Oct 31 03:45:18.732093 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.732070    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85" event={"ID":"1a742c43-144f-4fad-bd47-8e4f74e4d129","Type":"ContainerDied","Data":"6de555a326fd15bd7ee7a7a6eb8ad7d609236d830cbf29d14c87270862c7e375"}
Oct 31 03:45:18.732184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.732113    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"
Oct 31 03:45:18.735925 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.735897    4900 scope.go:122] "RemoveContainer" containerID="41ac00e5e10bd174bd291f5d20c2e216f8494b43a0230f6c7da46b219f2ad057"
Oct 31 03:45:18.747524 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.747505    4900 scope.go:122] "RemoveContainer" containerID="b1472fec89759ee49fc6443112da8d5e50bc71250f0193113ccd932a624fd818"
Oct 31 03:45:18.757996 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.757920    4900 scope.go:122] "RemoveContainer" containerID="d02ff9521085bfa03ec305f49b095535690acd6a9b302e9429688935ea78c5c1"
Oct 31 03:45:18.765623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.765605    4900 scope.go:122] "RemoveContainer" containerID="970f325228fdeb26cbd46e0dd78abc3d8747a6b48e2a54120c22e1b7c97d00bc"
Oct 31 03:45:18.930689 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:18.930658    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"]
Oct 31 03:45:19.039215 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.039182    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"]
Oct 31 03:45:19.045450 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.045407    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"]
Oct 31 03:45:19.048310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.048263    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-6f9fd2-29890305-mwmnb"]
Oct 31 03:45:19.155170 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.155116    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2047069d-6fdb-4e59-8b0f-c181883e8bf0" path="/var/lib/kubelet/pods/2047069d-6fdb-4e59-8b0f-c181883e8bf0/volumes"
Oct 31 03:45:19.156465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.156443    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"]
Oct 31 03:45:19.157753 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.157733    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c561e3-29890305-b2k85"]
Oct 31 03:45:19.264635 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.264602    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"]
Oct 31 03:45:19.291232 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.291170    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9360eb-29890305-ch5c9"]
Oct 31 03:45:19.740437 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.740409    4900 generic.go:356] "Generic (PLEG): container finished" podID="c481e920-e2ce-47e4-8664-1f938fffca1d" containerID="be2441309ca660689830d859ae9a5e93a038d8f760745766f2fcf12e10c85e42" exitCode=60
Oct 31 03:45:19.740941 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.740492    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g" event={"ID":"c481e920-e2ce-47e4-8664-1f938fffca1d","Type":"ContainerDied","Data":"be2441309ca660689830d859ae9a5e93a038d8f760745766f2fcf12e10c85e42"}
Oct 31 03:45:19.740941 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.740570    4900 scope.go:122] "RemoveContainer" containerID="cc1061c7dc72eccf86ee8f37d69d58c49604a8910c7317dca4a2ac0d296e18af"
Oct 31 03:45:19.741059 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.740961    4900 scope.go:122] "RemoveContainer" containerID="be2441309ca660689830d859ae9a5e93a038d8f760745766f2fcf12e10c85e42"
Oct 31 03:45:19.741275 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:19.741254    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g_odh-ai-gateway-infra(c481e920-e2ce-47e4-8664-1f938fffca1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g" podUID="c481e920-e2ce-47e4-8664-1f938fffca1d"
Oct 31 03:45:19.842163 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.842130    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"]
Oct 31 03:45:19.894590 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.894566    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"
Oct 31 03:45:19.925731 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:19.925708    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"
Oct 31 03:45:20.034506 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.034432    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c47366-f8f3-4b1e-9f55-b48c7cd551a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c47366-f8f3-4b1e-9f55-b48c7cd551a3-maas-api-service-ca\") pod \"28c47366-f8f3-4b1e-9f55-b48c7cd551a3\" (UID: \"28c47366-f8f3-4b1e-9f55-b48c7cd551a3\") "
Oct 31 03:45:20.034639 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.034615    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c47366-f8f3-4b1e-9f55-b48c7cd551a3-kube-api-access-4zj5w\" (UniqueName: \"kubernetes.io/projected/28c47366-f8f3-4b1e-9f55-b48c7cd551a3-kube-api-access-4zj5w\") pod \"28c47366-f8f3-4b1e-9f55-b48c7cd551a3\" (UID: \"28c47366-f8f3-4b1e-9f55-b48c7cd551a3\") "
Oct 31 03:45:20.034695 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.034646    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c90fdf27-108a-436f-808d-c41495a6fb8c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90fdf27-108a-436f-808d-c41495a6fb8c-maas-api-service-ca\") pod \"c90fdf27-108a-436f-808d-c41495a6fb8c\" (UID: \"c90fdf27-108a-436f-808d-c41495a6fb8c\") "
Oct 31 03:45:20.034695 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.034673    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c90fdf27-108a-436f-808d-c41495a6fb8c-kube-api-access-4ld9m\" (UniqueName: \"kubernetes.io/projected/c90fdf27-108a-436f-808d-c41495a6fb8c-kube-api-access-4ld9m\") pod \"c90fdf27-108a-436f-808d-c41495a6fb8c\" (UID: \"c90fdf27-108a-436f-808d-c41495a6fb8c\") "
Oct 31 03:45:20.034803 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.034745    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c47366-f8f3-4b1e-9f55-b48c7cd551a3-maas-api-service-ca" pod "28c47366-f8f3-4b1e-9f55-b48c7cd551a3" (UID: "28c47366-f8f3-4b1e-9f55-b48c7cd551a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:20.035078 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.035052    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90fdf27-108a-436f-808d-c41495a6fb8c-maas-api-service-ca" pod "c90fdf27-108a-436f-808d-c41495a6fb8c" (UID: "c90fdf27-108a-436f-808d-c41495a6fb8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:20.035271 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.035251    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28c47366-f8f3-4b1e-9f55-b48c7cd551a3-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:20.035321 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.035276    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90fdf27-108a-436f-808d-c41495a6fb8c-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:20.036825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.036763    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90fdf27-108a-436f-808d-c41495a6fb8c-kube-api-access-4ld9m" pod "c90fdf27-108a-436f-808d-c41495a6fb8c" (UID: "c90fdf27-108a-436f-808d-c41495a6fb8c"). InnerVolumeSpecName "kube-api-access-4ld9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:20.036939 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.036829    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c47366-f8f3-4b1e-9f55-b48c7cd551a3-kube-api-access-4zj5w" pod "28c47366-f8f3-4b1e-9f55-b48c7cd551a3" (UID: "28c47366-f8f3-4b1e-9f55-b48c7cd551a3"). InnerVolumeSpecName "kube-api-access-4zj5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:20.136546 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.136511    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zj5w\" (UniqueName: \"kubernetes.io/projected/28c47366-f8f3-4b1e-9f55-b48c7cd551a3-kube-api-access-4zj5w\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:20.136546 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.136545    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ld9m\" (UniqueName: \"kubernetes.io/projected/c90fdf27-108a-436f-808d-c41495a6fb8c-kube-api-access-4ld9m\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:20.754771 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.754742    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"
Oct 31 03:45:20.755221 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.754740    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8" event={"ID":"c90fdf27-108a-436f-808d-c41495a6fb8c","Type":"ContainerDied","Data":"2ebfd137c1c914ffab4c84b0766b643dc98d725dd75664c9d399cdf92fb9f0e0"}
Oct 31 03:45:20.755221 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.754927    4900 scope.go:122] "RemoveContainer" containerID="4284ef7d279427b0d1c12a6cc1900e26e5b3febb2876acaaf97f0163ad9f2123"
Oct 31 03:45:20.756280 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.756249    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk" event={"ID":"28c47366-f8f3-4b1e-9f55-b48c7cd551a3","Type":"ContainerDied","Data":"6d4f06f50a7c18c857565554ff5de493d2b9717433eafec6a535afec7ace019a"}
Oct 31 03:45:20.756431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.756264    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"
Oct 31 03:45:20.764568 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.764545    4900 scope.go:122] "RemoveContainer" containerID="47a2df5fe66cfd36d2b4864a4c694cbc1a9c44d8b5f7a13d15d466d679fe20f1"
Oct 31 03:45:20.910581 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.910552    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"
Oct 31 03:45:20.965218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.965187    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"]
Oct 31 03:45:20.970597 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:20.970573    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890305-s4qq8"]
Oct 31 03:45:21.048905 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.048830    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c481e920-e2ce-47e4-8664-1f938fffca1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c481e920-e2ce-47e4-8664-1f938fffca1d-maas-api-service-ca\") pod \"c481e920-e2ce-47e4-8664-1f938fffca1d\" (UID: \"c481e920-e2ce-47e4-8664-1f938fffca1d\") "
Oct 31 03:45:21.049026 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.048941    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c481e920-e2ce-47e4-8664-1f938fffca1d-kube-api-access-fnh8f\" (UniqueName: \"kubernetes.io/projected/c481e920-e2ce-47e4-8664-1f938fffca1d-kube-api-access-fnh8f\") pod \"c481e920-e2ce-47e4-8664-1f938fffca1d\" (UID: \"c481e920-e2ce-47e4-8664-1f938fffca1d\") "
Oct 31 03:45:21.049238 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.049210    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c481e920-e2ce-47e4-8664-1f938fffca1d-maas-api-service-ca" pod "c481e920-e2ce-47e4-8664-1f938fffca1d" (UID: "c481e920-e2ce-47e4-8664-1f938fffca1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:21.049517 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.049501    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c481e920-e2ce-47e4-8664-1f938fffca1d-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:21.051019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.051000    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c481e920-e2ce-47e4-8664-1f938fffca1d-kube-api-access-fnh8f" pod "c481e920-e2ce-47e4-8664-1f938fffca1d" (UID: "c481e920-e2ce-47e4-8664-1f938fffca1d"). InnerVolumeSpecName "kube-api-access-fnh8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:21.072334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.072305    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"]
Oct 31 03:45:21.077122 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.077100    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890305-5c5nk"]
Oct 31 03:45:21.150159 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.150125    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnh8f\" (UniqueName: \"kubernetes.io/projected/c481e920-e2ce-47e4-8664-1f938fffca1d-kube-api-access-fnh8f\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:21.154212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.154183    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a742c43-144f-4fad-bd47-8e4f74e4d129" path="/var/lib/kubelet/pods/1a742c43-144f-4fad-bd47-8e4f74e4d129/volumes"
Oct 31 03:45:21.154638 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.154618    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="214c92ee-f791-4685-bdee-19428df15788" path="/var/lib/kubelet/pods/214c92ee-f791-4685-bdee-19428df15788/volumes"
Oct 31 03:45:21.155071 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.155052    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3" path="/var/lib/kubelet/pods/28c47366-f8f3-4b1e-9f55-b48c7cd551a3/volumes"
Oct 31 03:45:21.155762 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.155742    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c90fdf27-108a-436f-808d-c41495a6fb8c" path="/var/lib/kubelet/pods/c90fdf27-108a-436f-808d-c41495a6fb8c/volumes"
Oct 31 03:45:21.764404 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.764372    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"
Oct 31 03:45:21.764404 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.764392    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g" event={"ID":"c481e920-e2ce-47e4-8664-1f938fffca1d","Type":"ContainerDied","Data":"31804cedad171a73ce95d8f6aaf2c3a1f48fd2fe2c386ef704a30c7a6a5dd248"}
Oct 31 03:45:21.764936 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.764460    4900 scope.go:122] "RemoveContainer" containerID="be2441309ca660689830d859ae9a5e93a038d8f760745766f2fcf12e10c85e42"
Oct 31 03:45:21.917019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.916983    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"]
Oct 31 03:45:21.919909 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:21.919886    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-03d737-29890305-knh9g"]
Oct 31 03:45:22.453598 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:22.453567    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:23.153113 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:23.153080    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c481e920-e2ce-47e4-8664-1f938fffca1d" path="/var/lib/kubelet/pods/c481e920-e2ce-47e4-8664-1f938fffca1d/volumes"
Oct 31 03:45:23.319169 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:23.319140    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:26.597089 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:26.597055    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:28.118880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:28.118842    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:32.129102 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:32.129068    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:32.382980 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:32.382895    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:32.761976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:32.761946    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:33.484013 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:33.483975    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:36.985741 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:45:36.985706    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:45:38.285330 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:38.285275    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:38.681264 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:38.681184    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:43.523303 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:43.523266    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:48.395456 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:48.395405    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:53.882381 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:53.882342    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:54.098582 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:54.098547    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:54.392607 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:54.392580    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-k2qfn_0b2b721d-414f-449f-ba75-918fb3a59972/payload-processing/0.log"
Oct 31 03:45:54.664242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:54.664156    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-k2qfn_0b2b721d-414f-449f-ba75-918fb3a59972/payload-processing/0.log"
Oct 31 03:45:55.079193 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:55.079158    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:55.324008 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:55.323976    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:55.565815 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:55.565749    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:55.785861 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:55.785830    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:56.030024 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:56.029993    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:56.262008 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:56.261974    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:45:59.809990 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:45:59.809951    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:00.050168 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:00.050131    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:00.321049 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:00.321008    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:01.226719 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:01.226691    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:01.458322 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:01.458283    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:03.047554 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.047493    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:03.273127 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.273089    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:03.519755 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.519714    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"]
Oct 31 03:46:03.520843 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.520774    4900 cpu_manager.go:414] "containerMap: removing container" podUID="214c92ee-f791-4685-bdee-19428df15788" containerName="cleanup"
Oct 31 03:46:03.521008 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.520985    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="214c92ee-f791-4685-bdee-19428df15788" containerName="cleanup"
Oct 31 03:46:03.521008 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521009    4900 cpu_manager.go:414] "containerMap: removing container" podUID="1a742c43-144f-4fad-bd47-8e4f74e4d129" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521016    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a742c43-144f-4fad-bd47-8e4f74e4d129" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521026    4900 cpu_manager.go:414] "containerMap: removing container" podUID="214c92ee-f791-4685-bdee-19428df15788" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521033    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="214c92ee-f791-4685-bdee-19428df15788" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521044    4900 cpu_manager.go:414] "containerMap: removing container" podUID="2047069d-6fdb-4e59-8b0f-c181883e8bf0" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521050    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2047069d-6fdb-4e59-8b0f-c181883e8bf0" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521064    4900 cpu_manager.go:414] "containerMap: removing container" podUID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521071    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521081    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c90fdf27-108a-436f-808d-c41495a6fb8c" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521089    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c90fdf27-108a-436f-808d-c41495a6fb8c" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521100    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c481e920-e2ce-47e4-8664-1f938fffca1d" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521106    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c481e920-e2ce-47e4-8664-1f938fffca1d" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521119    4900 cpu_manager.go:414] "containerMap: removing container" podUID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521126    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521144    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c90fdf27-108a-436f-808d-c41495a6fb8c" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521150    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c90fdf27-108a-436f-808d-c41495a6fb8c" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521172    4900 cpu_manager.go:414] "containerMap: removing container" podUID="1a742c43-144f-4fad-bd47-8e4f74e4d129" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521178    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a742c43-144f-4fad-bd47-8e4f74e4d129" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521211    4900 cpu_manager.go:414] "containerMap: removing container" podUID="2047069d-6fdb-4e59-8b0f-c181883e8bf0" containerName="cleanup"
Oct 31 03:46:03.521217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521217    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2047069d-6fdb-4e59-8b0f-c181883e8bf0" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521230    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c481e920-e2ce-47e4-8664-1f938fffca1d" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521237    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c481e920-e2ce-47e4-8664-1f938fffca1d" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521355    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c481e920-e2ce-47e4-8664-1f938fffca1d" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521368    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="214c92ee-f791-4685-bdee-19428df15788" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521376    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a742c43-144f-4fad-bd47-8e4f74e4d129" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521383    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="214c92ee-f791-4685-bdee-19428df15788" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521393    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="2047069d-6fdb-4e59-8b0f-c181883e8bf0" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521403    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521410    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c90fdf27-108a-436f-808d-c41495a6fb8c" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521418    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c481e920-e2ce-47e4-8664-1f938fffca1d" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521425    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521434    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c90fdf27-108a-436f-808d-c41495a6fb8c" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521441    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a742c43-144f-4fad-bd47-8e4f74e4d129" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521449    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c90fdf27-108a-436f-808d-c41495a6fb8c" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521457    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c481e920-e2ce-47e4-8664-1f938fffca1d" containerName="cleanup"
Oct 31 03:46:03.522226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.521468    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="2047069d-6fdb-4e59-8b0f-c181883e8bf0" containerName="cleanup"
Oct 31 03:46:03.524761 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.524735    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"
Oct 31 03:46:03.574448 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.574410    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-fc85c7\""
Oct 31 03:46:03.620648 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.620620    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-fc85c7-dockercfg-vjz95\""
Oct 31 03:46:03.622980 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.622957    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"]
Oct 31 03:46:03.654588 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.654553    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdshr\" (UniqueName: \"kubernetes.io/projected/3942dcff-b885-43c1-bee5-bfca753b1df0-kube-api-access-tdshr\") pod \"payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2\" (UID: \"3942dcff-b885-43c1-bee5-bfca753b1df0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"
Oct 31 03:46:03.654781 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.654691    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3942dcff-b885-43c1-bee5-bfca753b1df0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2\" (UID: \"3942dcff-b885-43c1-bee5-bfca753b1df0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"
Oct 31 03:46:03.692430 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.692393    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"]
Oct 31 03:46:03.693140 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693119    4900 cpu_manager.go:414] "containerMap: removing container" podUID="1a742c43-144f-4fad-bd47-8e4f74e4d129" containerName="cleanup"
Oct 31 03:46:03.693140 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693140    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a742c43-144f-4fad-bd47-8e4f74e4d129" containerName="cleanup"
Oct 31 03:46:03.693294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693172    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c481e920-e2ce-47e4-8664-1f938fffca1d" containerName="cleanup"
Oct 31 03:46:03.693294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693178    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c481e920-e2ce-47e4-8664-1f938fffca1d" containerName="cleanup"
Oct 31 03:46:03.693294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693190    4900 cpu_manager.go:414] "containerMap: removing container" podUID="214c92ee-f791-4685-bdee-19428df15788" containerName="cleanup"
Oct 31 03:46:03.693294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693195    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="214c92ee-f791-4685-bdee-19428df15788" containerName="cleanup"
Oct 31 03:46:03.693294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693201    4900 cpu_manager.go:414] "containerMap: removing container" podUID="2047069d-6fdb-4e59-8b0f-c181883e8bf0" containerName="cleanup"
Oct 31 03:46:03.693294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693206    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2047069d-6fdb-4e59-8b0f-c181883e8bf0" containerName="cleanup"
Oct 31 03:46:03.693294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693216    4900 cpu_manager.go:414] "containerMap: removing container" podUID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3" containerName="cleanup"
Oct 31 03:46:03.693294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693221    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3" containerName="cleanup"
Oct 31 03:46:03.693294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693234    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c90fdf27-108a-436f-808d-c41495a6fb8c" containerName="cleanup"
Oct 31 03:46:03.693294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693238    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c90fdf27-108a-436f-808d-c41495a6fb8c" containerName="cleanup"
Oct 31 03:46:03.693762 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693336    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="2047069d-6fdb-4e59-8b0f-c181883e8bf0" containerName="cleanup"
Oct 31 03:46:03.693762 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693346    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="214c92ee-f791-4685-bdee-19428df15788" containerName="cleanup"
Oct 31 03:46:03.693762 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693354    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c47366-f8f3-4b1e-9f55-b48c7cd551a3" containerName="cleanup"
Oct 31 03:46:03.693762 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.693363    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a742c43-144f-4fad-bd47-8e4f74e4d129" containerName="cleanup"
Oct 31 03:46:03.695890 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.695872    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"
Oct 31 03:46:03.745935 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.745904    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"]
Oct 31 03:46:03.756511 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.756473    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdshr\" (UniqueName: \"kubernetes.io/projected/3942dcff-b885-43c1-bee5-bfca753b1df0-kube-api-access-tdshr\") pod \"payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2\" (UID: \"3942dcff-b885-43c1-bee5-bfca753b1df0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"
Oct 31 03:46:03.756710 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.756580    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ded06c-f133-49af-9ed0-7dc31f49e95c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp\" (UID: \"60ded06c-f133-49af-9ed0-7dc31f49e95c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"
Oct 31 03:46:03.756710 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.756635    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3942dcff-b885-43c1-bee5-bfca753b1df0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2\" (UID: \"3942dcff-b885-43c1-bee5-bfca753b1df0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"
Oct 31 03:46:03.756867 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.756772    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnfpz\" (UniqueName: \"kubernetes.io/projected/60ded06c-f133-49af-9ed0-7dc31f49e95c-kube-api-access-bnfpz\") pod \"payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp\" (UID: \"60ded06c-f133-49af-9ed0-7dc31f49e95c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"
Oct 31 03:46:03.757407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.757388    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3942dcff-b885-43c1-bee5-bfca753b1df0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2\" (UID: \"3942dcff-b885-43c1-bee5-bfca753b1df0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"
Oct 31 03:46:03.809528 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.809460    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdshr\" (UniqueName: \"kubernetes.io/projected/3942dcff-b885-43c1-bee5-bfca753b1df0-kube-api-access-tdshr\") pod \"payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2\" (UID: \"3942dcff-b885-43c1-bee5-bfca753b1df0\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"
Oct 31 03:46:03.845509 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.845470    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"
Oct 31 03:46:03.857585 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.857554    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ded06c-f133-49af-9ed0-7dc31f49e95c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp\" (UID: \"60ded06c-f133-49af-9ed0-7dc31f49e95c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"
Oct 31 03:46:03.857694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.857639    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnfpz\" (UniqueName: \"kubernetes.io/projected/60ded06c-f133-49af-9ed0-7dc31f49e95c-kube-api-access-bnfpz\") pod \"payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp\" (UID: \"60ded06c-f133-49af-9ed0-7dc31f49e95c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"
Oct 31 03:46:03.858283 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.858264    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ded06c-f133-49af-9ed0-7dc31f49e95c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp\" (UID: \"60ded06c-f133-49af-9ed0-7dc31f49e95c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"
Oct 31 03:46:03.916756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:03.916723    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnfpz\" (UniqueName: \"kubernetes.io/projected/60ded06c-f133-49af-9ed0-7dc31f49e95c-kube-api-access-bnfpz\") pod \"payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp\" (UID: \"60ded06c-f133-49af-9ed0-7dc31f49e95c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"
Oct 31 03:46:04.009019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:04.008983    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"
Oct 31 03:46:04.215584 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:04.215555    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"]
Oct 31 03:46:04.217091 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:46:04.217065    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3942dcff_b885_43c1_bee5_bfca753b1df0.slice/crio-fcac57cbdf97cca8f0a35a091e6aa3dabc00086da40878b0dd393199d247fb57 WatchSource:0}: Error finding container fcac57cbdf97cca8f0a35a091e6aa3dabc00086da40878b0dd393199d247fb57: Status 404 returned error can't find the container with id fcac57cbdf97cca8f0a35a091e6aa3dabc00086da40878b0dd393199d247fb57
Oct 31 03:46:04.375686 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:04.375653    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"]
Oct 31 03:46:04.377160 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:46:04.377126    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ded06c_f133_49af_9ed0_7dc31f49e95c.slice/crio-81b6668226d84fb53109bd36b4ff0cf8df4e9e90ab4290b370bef2a1857d25ac WatchSource:0}: Error finding container 81b6668226d84fb53109bd36b4ff0cf8df4e9e90ab4290b370bef2a1857d25ac: Status 404 returned error can't find the container with id 81b6668226d84fb53109bd36b4ff0cf8df4e9e90ab4290b370bef2a1857d25ac
Oct 31 03:46:05.001290 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:05.001242    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp" event={"ID":"60ded06c-f133-49af-9ed0-7dc31f49e95c","Type":"ContainerStarted","Data":"0748210bf7f2e751820bf84407347b7a9bdb7eb27a12006a8eee18d67f127453"}
Oct 31 03:46:05.001480 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:05.001310    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp" event={"ID":"60ded06c-f133-49af-9ed0-7dc31f49e95c","Type":"ContainerStarted","Data":"81b6668226d84fb53109bd36b4ff0cf8df4e9e90ab4290b370bef2a1857d25ac"}
Oct 31 03:46:05.001480 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:05.001400    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"
Oct 31 03:46:05.002976 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:05.002948    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2" event={"ID":"3942dcff-b885-43c1-bee5-bfca753b1df0","Type":"ContainerStarted","Data":"6d0d30e9a6de1964d5ff9491390114ff9eadfae0f6426472b5131387fd8721df"}
Oct 31 03:46:05.003106 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:05.002994    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"
Oct 31 03:46:05.003106 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:05.003013    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2" event={"ID":"3942dcff-b885-43c1-bee5-bfca753b1df0","Type":"ContainerStarted","Data":"fcac57cbdf97cca8f0a35a091e6aa3dabc00086da40878b0dd393199d247fb57"}
Oct 31 03:46:05.110436 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:05.110380    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp" podStartSLOduration=2.110364102 podStartE2EDuration="2.110364102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:05.107952078 +0000 UTC m=+2318.074285438" watchObservedRunningTime="2026-10-31 03:46:05.110364102 +0000 UTC m=+2318.076697446"
Oct 31 03:46:05.217472 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:05.217405    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2" podStartSLOduration=2.21738921 podStartE2EDuration="2.21738921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:05.215274758 +0000 UTC m=+2318.181608104" watchObservedRunningTime="2026-10-31 03:46:05.21738921 +0000 UTC m=+2318.183722555"
Oct 31 03:46:06.030711 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:06.030673    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:06.825580 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:06.825543    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:06.984591 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:46:06.984556    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:46:07.059977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:07.059949    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:07.291533 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:07.291500    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:11.639622 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:11.639585    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:12.307388 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:12.307351    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:12.606284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:12.606192    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:12.829907 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:12.829874    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:16.021313 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:16.021284    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"
Oct 31 03:46:16.021678 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:16.021350    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"
Oct 31 03:46:17.008486 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:17.008450    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:18.895210 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:18.895176    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:23.899147 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:23.899117    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:27.033358 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:27.033321    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:27.253939 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:27.253905    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:28.731400 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:28.731362    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:29.297209 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:29.297172    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:30.767374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:30.767338    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:35.200645 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:35.200607    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:35.688572 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:35.688538    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:36.986482 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:46:36.986446    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:46:38.092606 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:38.092572    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:38.696096 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:38.696061    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:40.155779 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:40.155700    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:41.217065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:41.217031    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:41.918996 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:41.918966    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:42.141735 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:42.141692    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:43.275235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:43.275192    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:43.502297 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:43.502266    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:44.469887 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:44.469845    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:44.821901 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:44.821864    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:45.366617 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:45.366585    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:45.610981 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:45.610950    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:46.176771 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:46.176720    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:46.828758 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:46.828727    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:47.377880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:47.377843    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:48.427524 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:48.427489    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:50.245796 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:50.245757    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:51.756033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:51.755994    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:52.249439 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:52.249412    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:53.566749 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:53.566713    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:55.495410 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:55.495372    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:57.326639 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:57.326604    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:57.647146 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:57.647026    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:58.550119 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:58.550088    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:59.095706 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:59.095674    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:46:59.319047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:46:59.319009    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:00.334942 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:00.334911    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:00.541614 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:00.541585    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:00.761157 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:00.761122    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:03.511209 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:03.511173    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:03.722974 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:03.722940    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:05.087461 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:05.087427    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:05.320911 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:05.320880    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:05.706545 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:05.706514    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:05.924048 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:05.924003    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:06.984668 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:47:06.984623    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:47:07.378015 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:07.377921    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:07.967494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:07.967459    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:08.807954 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:08.807922    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:09.811376 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:09.811341    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:10.314156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:10.314096    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:13.289237 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:13.289185    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:14.701983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:14.701944    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:18.522080 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:18.522034    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:23.411412 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:23.411373    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:23.629939 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:23.629903    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:25.663224 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:25.663187    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:26.273986 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:26.273922    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:26.510375 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:26.510344    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:27.760959 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:27.760920    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:28.092950 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.092870    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"]
Oct 31 03:47:28.093125 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.093104    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2" podUID="c72b34a0-8246-4c13-a3a3-49501c50d6e5" containerName="payload-pre-processing" containerID="cri-o://0dc5a751e02deddf8c003200ad5d989b637536e5ca41c0ebf72e481393f1e952" gracePeriod=30
Oct 31 03:47:28.266147 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.266110    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:28.371288 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.371262    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"
Oct 31 03:47:28.432590 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.432559    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c72b34a0-8246-4c13-a3a3-49501c50d6e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72b34a0-8246-4c13-a3a3-49501c50d6e5-plugins-config-volume\") pod \"c72b34a0-8246-4c13-a3a3-49501c50d6e5\" (UID: \"c72b34a0-8246-4c13-a3a3-49501c50d6e5\") "
Oct 31 03:47:28.432752 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.432661    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72b34a0-8246-4c13-a3a3-49501c50d6e5-kube-api-access-rkpfx\" (UniqueName: \"kubernetes.io/projected/c72b34a0-8246-4c13-a3a3-49501c50d6e5-kube-api-access-rkpfx\") pod \"c72b34a0-8246-4c13-a3a3-49501c50d6e5\" (UID: \"c72b34a0-8246-4c13-a3a3-49501c50d6e5\") "
Oct 31 03:47:28.432959 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.432937    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72b34a0-8246-4c13-a3a3-49501c50d6e5-plugins-config-volume" pod "c72b34a0-8246-4c13-a3a3-49501c50d6e5" (UID: "c72b34a0-8246-4c13-a3a3-49501c50d6e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:28.433182 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.433168    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72b34a0-8246-4c13-a3a3-49501c50d6e5-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:28.434696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.434669    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72b34a0-8246-4c13-a3a3-49501c50d6e5-kube-api-access-rkpfx" pod "c72b34a0-8246-4c13-a3a3-49501c50d6e5" (UID: "c72b34a0-8246-4c13-a3a3-49501c50d6e5"). InnerVolumeSpecName "kube-api-access-rkpfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:28.471211 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.471178    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:28.503012 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.502981    4900 generic.go:356] "Generic (PLEG): container finished" podID="c72b34a0-8246-4c13-a3a3-49501c50d6e5" containerID="0dc5a751e02deddf8c003200ad5d989b637536e5ca41c0ebf72e481393f1e952" exitCode=0
Oct 31 03:47:28.503160 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.503033    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2" event={"ID":"c72b34a0-8246-4c13-a3a3-49501c50d6e5","Type":"ContainerDied","Data":"0dc5a751e02deddf8c003200ad5d989b637536e5ca41c0ebf72e481393f1e952"}
Oct 31 03:47:28.503160 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.503042    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"
Oct 31 03:47:28.503160 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.503108    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2" event={"ID":"c72b34a0-8246-4c13-a3a3-49501c50d6e5","Type":"ContainerDied","Data":"933e09955772ce60fb2364b940eb6199841cfa64f81aa03a41f0f13246442641"}
Oct 31 03:47:28.503160 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.503132    4900 scope.go:122] "RemoveContainer" containerID="0dc5a751e02deddf8c003200ad5d989b637536e5ca41c0ebf72e481393f1e952"
Oct 31 03:47:28.514037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.514017    4900 scope.go:122] "RemoveContainer" containerID="0dc5a751e02deddf8c003200ad5d989b637536e5ca41c0ebf72e481393f1e952"
Oct 31 03:47:28.514310 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:47:28.514290    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc5a751e02deddf8c003200ad5d989b637536e5ca41c0ebf72e481393f1e952\": container with ID starting with 0dc5a751e02deddf8c003200ad5d989b637536e5ca41c0ebf72e481393f1e952 not found: ID does not exist" containerID="0dc5a751e02deddf8c003200ad5d989b637536e5ca41c0ebf72e481393f1e952"
Oct 31 03:47:28.514393 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.514319    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc5a751e02deddf8c003200ad5d989b637536e5ca41c0ebf72e481393f1e952"} err="failed to get container status \"0dc5a751e02deddf8c003200ad5d989b637536e5ca41c0ebf72e481393f1e952\": rpc error: code = NotFound desc = could not find container \"0dc5a751e02deddf8c003200ad5d989b637536e5ca41c0ebf72e481393f1e952\": container with ID starting with 0dc5a751e02deddf8c003200ad5d989b637536e5ca41c0ebf72e481393f1e952 not found: ID does not exist"
Oct 31 03:47:28.534102 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.534075    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkpfx\" (UniqueName: \"kubernetes.io/projected/c72b34a0-8246-4c13-a3a3-49501c50d6e5-kube-api-access-rkpfx\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:28.625886 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.625808    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:47:28.650246 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.650222    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:47:28.668935 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.668909    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"]
Oct 31 03:47:28.678225 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:28.678198    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c561e3-647fb7855f-bfjm2"]
Oct 31 03:47:29.000310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:29.000280    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:29.152231 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:29.152198    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72b34a0-8246-4c13-a3a3-49501c50d6e5" path="/var/lib/kubelet/pods/c72b34a0-8246-4c13-a3a3-49501c50d6e5/volumes"
Oct 31 03:47:30.966088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:30.966043    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:31.194465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:31.194427    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:34.116116 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:34.115490    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:34.351224 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:34.351185    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:36.030316 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:36.030275    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:36.262156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:36.262123    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:36.603559 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:36.603523    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:36.988107 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:47:36.988065    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:47:38.192235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:38.192205    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:38.422894 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:38.422855    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:38.606889 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:38.606859    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"]
Oct 31 03:47:38.607118 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:38.607092    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerName="istio-proxy" containerID="cri-o://3c1f59996c843325b3b3bf55df04542a14b8b39207cfe1f7f05fe9408923b137" gracePeriod=30
Oct 31 03:47:38.689927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:38.689888    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:39.599824 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:39.599770    4900 patch_prober.go:29] interesting pod/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:47:39.600196 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:39.599844    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:47:39.930198 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:39.930115    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:40.432403 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:40.432367    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:42.032909 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:42.032875    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:42.262348 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:42.262316    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:42.762213 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:42.762182    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:43.060864 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.060774    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:43.857101 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.857078    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:47:43.952009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.951938    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-workload-socket\") pod \"5f07fc2e-b24b-46c6-9571-39282e304a53\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") "
Oct 31 03:47:43.952009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.951980    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f07fc2e-b24b-46c6-9571-39282e304a53-kube-api-access-l282z\" (UniqueName: \"kubernetes.io/projected/5f07fc2e-b24b-46c6-9571-39282e304a53-kube-api-access-l282z\") pod \"5f07fc2e-b24b-46c6-9571-39282e304a53\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") "
Oct 31 03:47:43.952240 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.952017    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-credential-socket\") pod \"5f07fc2e-b24b-46c6-9571-39282e304a53\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") "
Oct 31 03:47:43.952240 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.952077    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f07fc2e-b24b-46c6-9571-39282e304a53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f07fc2e-b24b-46c6-9571-39282e304a53-istiod-ca-cert\") pod \"5f07fc2e-b24b-46c6-9571-39282e304a53\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") "
Oct 31 03:47:43.952240 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.952160    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-workload-certs\") pod \"5f07fc2e-b24b-46c6-9571-39282e304a53\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") "
Oct 31 03:47:43.952240 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.952166    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-workload-socket" pod "5f07fc2e-b24b-46c6-9571-39282e304a53" (UID: "5f07fc2e-b24b-46c6-9571-39282e304a53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:43.952471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.952276    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-podinfo\") pod \"5f07fc2e-b24b-46c6-9571-39282e304a53\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") "
Oct 31 03:47:43.952471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.952314    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-token\" (UniqueName: \"kubernetes.io/projected/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-token\") pod \"5f07fc2e-b24b-46c6-9571-39282e304a53\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") "
Oct 31 03:47:43.952471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.952350    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-envoy\") pod \"5f07fc2e-b24b-46c6-9571-39282e304a53\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") "
Oct 31 03:47:43.952471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.952378    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-data\") pod \"5f07fc2e-b24b-46c6-9571-39282e304a53\" (UID: \"5f07fc2e-b24b-46c6-9571-39282e304a53\") "
Oct 31 03:47:43.952471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.952366    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-credential-socket" pod "5f07fc2e-b24b-46c6-9571-39282e304a53" (UID: "5f07fc2e-b24b-46c6-9571-39282e304a53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:43.952471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.952442    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-workload-certs" pod "5f07fc2e-b24b-46c6-9571-39282e304a53" (UID: "5f07fc2e-b24b-46c6-9571-39282e304a53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:43.952783 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.952532    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f07fc2e-b24b-46c6-9571-39282e304a53-istiod-ca-cert" pod "5f07fc2e-b24b-46c6-9571-39282e304a53" (UID: "5f07fc2e-b24b-46c6-9571-39282e304a53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:43.952783 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.952736    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-data" pod "5f07fc2e-b24b-46c6-9571-39282e304a53" (UID: "5f07fc2e-b24b-46c6-9571-39282e304a53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:43.953016 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.952996    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f07fc2e-b24b-46c6-9571-39282e304a53-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:43.953079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.953024    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:43.953079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.953038    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:43.953079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.953055    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:43.953079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.953070    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:43.954408 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.954377    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-podinfo" pod "5f07fc2e-b24b-46c6-9571-39282e304a53" (UID: "5f07fc2e-b24b-46c6-9571-39282e304a53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:47:43.954408 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.954398    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f07fc2e-b24b-46c6-9571-39282e304a53-kube-api-access-l282z" pod "5f07fc2e-b24b-46c6-9571-39282e304a53" (UID: "5f07fc2e-b24b-46c6-9571-39282e304a53"). InnerVolumeSpecName "kube-api-access-l282z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:43.954709 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.954691    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-envoy" pod "5f07fc2e-b24b-46c6-9571-39282e304a53" (UID: "5f07fc2e-b24b-46c6-9571-39282e304a53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:43.954840 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:43.954719    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-token" pod "5f07fc2e-b24b-46c6-9571-39282e304a53" (UID: "5f07fc2e-b24b-46c6-9571-39282e304a53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:44.054407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.054361    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:44.054407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.054402    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:44.054407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.054411    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f07fc2e-b24b-46c6-9571-39282e304a53-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:44.054407 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.054421    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l282z\" (UniqueName: \"kubernetes.io/projected/5f07fc2e-b24b-46c6-9571-39282e304a53-kube-api-access-l282z\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:44.602702 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.602670    4900 generic.go:356] "Generic (PLEG): container finished" podID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerID="3c1f59996c843325b3b3bf55df04542a14b8b39207cfe1f7f05fe9408923b137" exitCode=0
Oct 31 03:47:44.603211 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.602712    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" event={"ID":"5f07fc2e-b24b-46c6-9571-39282e304a53","Type":"ContainerDied","Data":"3c1f59996c843325b3b3bf55df04542a14b8b39207cfe1f7f05fe9408923b137"}
Oct 31 03:47:44.603211 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.602720    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"
Oct 31 03:47:44.603211 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.602766    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n" event={"ID":"5f07fc2e-b24b-46c6-9571-39282e304a53","Type":"ContainerDied","Data":"63e0ff25d0f09b3d6a0b2447871f1c68f875c4aca6911c992a00a567f4ce4f9a"}
Oct 31 03:47:44.603211 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.602813    4900 scope.go:122] "RemoveContainer" containerID="3c1f59996c843325b3b3bf55df04542a14b8b39207cfe1f7f05fe9408923b137"
Oct 31 03:47:44.615535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.615515    4900 scope.go:122] "RemoveContainer" containerID="92be4990e1d9c5cd34a4858711be9fd8dac16d003606ce688212a0f98261e67a"
Oct 31 03:47:44.624236 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.624212    4900 scope.go:122] "RemoveContainer" containerID="3c1f59996c843325b3b3bf55df04542a14b8b39207cfe1f7f05fe9408923b137"
Oct 31 03:47:44.624492 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:47:44.624471    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1f59996c843325b3b3bf55df04542a14b8b39207cfe1f7f05fe9408923b137\": container with ID starting with 3c1f59996c843325b3b3bf55df04542a14b8b39207cfe1f7f05fe9408923b137 not found: ID does not exist" containerID="3c1f59996c843325b3b3bf55df04542a14b8b39207cfe1f7f05fe9408923b137"
Oct 31 03:47:44.624571 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.624504    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1f59996c843325b3b3bf55df04542a14b8b39207cfe1f7f05fe9408923b137"} err="failed to get container status \"3c1f59996c843325b3b3bf55df04542a14b8b39207cfe1f7f05fe9408923b137\": rpc error: code = NotFound desc = could not find container \"3c1f59996c843325b3b3bf55df04542a14b8b39207cfe1f7f05fe9408923b137\": container with ID starting with 3c1f59996c843325b3b3bf55df04542a14b8b39207cfe1f7f05fe9408923b137 not found: ID does not exist"
Oct 31 03:47:44.624571 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.624533    4900 scope.go:122] "RemoveContainer" containerID="92be4990e1d9c5cd34a4858711be9fd8dac16d003606ce688212a0f98261e67a"
Oct 31 03:47:44.624851 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:47:44.624825    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92be4990e1d9c5cd34a4858711be9fd8dac16d003606ce688212a0f98261e67a\": container with ID starting with 92be4990e1d9c5cd34a4858711be9fd8dac16d003606ce688212a0f98261e67a not found: ID does not exist" containerID="92be4990e1d9c5cd34a4858711be9fd8dac16d003606ce688212a0f98261e67a"
Oct 31 03:47:44.624904 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.624865    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92be4990e1d9c5cd34a4858711be9fd8dac16d003606ce688212a0f98261e67a"} err="failed to get container status \"92be4990e1d9c5cd34a4858711be9fd8dac16d003606ce688212a0f98261e67a\": rpc error: code = NotFound desc = could not find container \"92be4990e1d9c5cd34a4858711be9fd8dac16d003606ce688212a0f98261e67a\": container with ID starting with 92be4990e1d9c5cd34a4858711be9fd8dac16d003606ce688212a0f98261e67a not found: ID does not exist"
Oct 31 03:47:44.766690 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.766661    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"]
Oct 31 03:47:44.816483 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:44.816455    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-03d737-openshift-default-599d9589d5-gbd9n"]
Oct 31 03:47:45.152494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:45.152456    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" path="/var/lib/kubelet/pods/5f07fc2e-b24b-46c6-9571-39282e304a53/volumes"
Oct 31 03:47:46.993397 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:46.993363    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:47.230405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:47.230375    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:47.587394 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:47.587361    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:52.674936 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:52.674888    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:52.900520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:52.900478    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:53.121157 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:53.121115    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:55.841487 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:55.841440    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:47:58.271294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:47:58.271256    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:02.633769 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:02.633725    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:05.481272 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:05.481234    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:06.010292 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:06.010261    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:06.984028 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:48:06.983992    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:48:10.589238 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:10.589148    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:12.265402 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:12.265371    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:12.476916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:12.476870    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:12.684091 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:12.683991    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:15.853260 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:15.853229    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:18.016393 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:18.016354    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:20.611617 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:20.611585    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:20.821502 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:20.821469    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:21.031317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:21.031272    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:25.694606 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:25.694575    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:30.595743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:30.595699    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:35.493448 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:35.493404    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:36.985180 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:48:36.985142    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:48:40.459490 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:40.459456    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:40.677803 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:40.677758    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:44.931056 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:44.931017    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:46.861084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:46.861028    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:48.477291 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:48.477258    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"]
Oct 31 03:48:48.477710 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:48.477483    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4" podUID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerName="istio-proxy" containerID="cri-o://ba93d0fd0f3edcfc90bc1e8d597e76e466d068f3041b752c6bf307d90018cfd9" gracePeriod=30
Oct 31 03:48:48.527335 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:48.527303    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"]
Oct 31 03:48:48.527565 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:48.527534    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n" podUID="dcc714a7-95eb-4aab-9974-09e37b18e6c9" containerName="istio-proxy" containerID="cri-o://c954017a5e176be6944ea4718890da619129645c57257b42eca514fa5458bdcc" gracePeriod=30
Oct 31 03:48:48.958451 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:48.958415    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:48:48.958641 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:48.958467    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n" podUID="dcc714a7-95eb-4aab-9974-09e37b18e6c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:48:50.465242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:50.465207    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:50.664540 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:50.664507    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:52.123028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:52.122979    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:52.340111 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:52.340018    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:52.637714 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:52.637634    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:53.441972 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.441937    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:53.829981 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.829955    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:48:53.833933 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.833909    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:48:53.916608 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.916575    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc714a7-95eb-4aab-9974-09e37b18e6c9-kube-api-access-9pj22\" (UniqueName: \"kubernetes.io/projected/dcc714a7-95eb-4aab-9974-09e37b18e6c9-kube-api-access-9pj22\") pod \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") "
Oct 31 03:48:53.916835 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.916627    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-workload-socket\") pod \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") "
Oct 31 03:48:53.916835 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.916678    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-credential-socket\") pod \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") "
Oct 31 03:48:53.916835 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.916719    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-envoy\") pod \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") "
Oct 31 03:48:53.916835 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.916753    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istiod-ca-cert\") pod \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") "
Oct 31 03:48:53.916835 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.916796    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-workload-certs\") pod \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") "
Oct 31 03:48:53.916835 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.916830    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-podinfo\") pod \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") "
Oct 31 03:48:53.917173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.916868    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-workload-socket\") pod \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") "
Oct 31 03:48:53.917173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.916901    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-data\") pod \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") "
Oct 31 03:48:53.917173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.916959    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istiod-ca-cert\") pod \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") "
Oct 31 03:48:53.917173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.916981    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-workload-socket" pod "a63cf118-4a57-48ba-852b-bcaa10da6d4b" (UID: "a63cf118-4a57-48ba-852b-bcaa10da6d4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:53.917173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917001    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-workload-certs\") pod \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") "
Oct 31 03:48:53.917173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917014    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-credential-socket" pod "a63cf118-4a57-48ba-852b-bcaa10da6d4b" (UID: "a63cf118-4a57-48ba-852b-bcaa10da6d4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:53.917173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917045    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-token\" (UniqueName: \"kubernetes.io/projected/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-token\") pod \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") "
Oct 31 03:48:53.917173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917087    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-data\") pod \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") "
Oct 31 03:48:53.917173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917116    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-credential-socket\") pod \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") "
Oct 31 03:48:53.917173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917160    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-envoy\") pod \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") "
Oct 31 03:48:53.917696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917172    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-workload-certs" pod "dcc714a7-95eb-4aab-9974-09e37b18e6c9" (UID: "dcc714a7-95eb-4aab-9974-09e37b18e6c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:53.917696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917217    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-workload-socket" pod "dcc714a7-95eb-4aab-9974-09e37b18e6c9" (UID: "dcc714a7-95eb-4aab-9974-09e37b18e6c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:53.917696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917248    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-podinfo\") pod \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") "
Oct 31 03:48:53.917696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917280    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-token\" (UniqueName: \"kubernetes.io/projected/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-token\") pod \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\" (UID: \"dcc714a7-95eb-4aab-9974-09e37b18e6c9\") "
Oct 31 03:48:53.917696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917286    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istiod-ca-cert" pod "dcc714a7-95eb-4aab-9974-09e37b18e6c9" (UID: "dcc714a7-95eb-4aab-9974-09e37b18e6c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:53.917696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917308    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a63cf118-4a57-48ba-852b-bcaa10da6d4b-kube-api-access-rhn96\" (UniqueName: \"kubernetes.io/projected/a63cf118-4a57-48ba-852b-bcaa10da6d4b-kube-api-access-rhn96\") pod \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\" (UID: \"a63cf118-4a57-48ba-852b-bcaa10da6d4b\") "
Oct 31 03:48:53.917696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917425    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-data" pod "a63cf118-4a57-48ba-852b-bcaa10da6d4b" (UID: "a63cf118-4a57-48ba-852b-bcaa10da6d4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:53.917696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917437    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-data" pod "dcc714a7-95eb-4aab-9974-09e37b18e6c9" (UID: "dcc714a7-95eb-4aab-9974-09e37b18e6c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:53.917696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917442    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istiod-ca-cert" pod "a63cf118-4a57-48ba-852b-bcaa10da6d4b" (UID: "a63cf118-4a57-48ba-852b-bcaa10da6d4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:53.917696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917638    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-workload-certs" pod "a63cf118-4a57-48ba-852b-bcaa10da6d4b" (UID: "a63cf118-4a57-48ba-852b-bcaa10da6d4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:53.918268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917938    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:53.918268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917964    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:53.918268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917980    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:53.918268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.917995    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:53.918268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.918009    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:53.918268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.918023    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:53.918268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.918038    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:53.918268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.918051    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:53.918268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.918067    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:53.918268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.918205    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-credential-socket" pod "dcc714a7-95eb-4aab-9974-09e37b18e6c9" (UID: "dcc714a7-95eb-4aab-9974-09e37b18e6c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:53.920187 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.920157    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-envoy" pod "a63cf118-4a57-48ba-852b-bcaa10da6d4b" (UID: "a63cf118-4a57-48ba-852b-bcaa10da6d4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:53.920187 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.920164    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc714a7-95eb-4aab-9974-09e37b18e6c9-kube-api-access-9pj22" pod "dcc714a7-95eb-4aab-9974-09e37b18e6c9" (UID: "dcc714a7-95eb-4aab-9974-09e37b18e6c9"). InnerVolumeSpecName "kube-api-access-9pj22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:53.920384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.920214    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-podinfo" pod "dcc714a7-95eb-4aab-9974-09e37b18e6c9" (UID: "dcc714a7-95eb-4aab-9974-09e37b18e6c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:48:53.920384 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.920254    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63cf118-4a57-48ba-852b-bcaa10da6d4b-kube-api-access-rhn96" pod "a63cf118-4a57-48ba-852b-bcaa10da6d4b" (UID: "a63cf118-4a57-48ba-852b-bcaa10da6d4b"). InnerVolumeSpecName "kube-api-access-rhn96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:53.920505 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.920439    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-token" pod "a63cf118-4a57-48ba-852b-bcaa10da6d4b" (UID: "a63cf118-4a57-48ba-852b-bcaa10da6d4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:53.920593 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.920570    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-podinfo" pod "a63cf118-4a57-48ba-852b-bcaa10da6d4b" (UID: "a63cf118-4a57-48ba-852b-bcaa10da6d4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:48:53.920925 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.920906    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-token" pod "dcc714a7-95eb-4aab-9974-09e37b18e6c9" (UID: "dcc714a7-95eb-4aab-9974-09e37b18e6c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:53.920996 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.920966    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-envoy" pod "dcc714a7-95eb-4aab-9974-09e37b18e6c9" (UID: "dcc714a7-95eb-4aab-9974-09e37b18e6c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:53.992229 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.992189    4900 generic.go:356] "Generic (PLEG): container finished" podID="dcc714a7-95eb-4aab-9974-09e37b18e6c9" containerID="c954017a5e176be6944ea4718890da619129645c57257b42eca514fa5458bdcc" exitCode=0
Oct 31 03:48:53.992391 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.992256    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"
Oct 31 03:48:53.992391 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.992249    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n" event={"ID":"dcc714a7-95eb-4aab-9974-09e37b18e6c9","Type":"ContainerDied","Data":"c954017a5e176be6944ea4718890da619129645c57257b42eca514fa5458bdcc"}
Oct 31 03:48:53.992391 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.992384    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n" event={"ID":"dcc714a7-95eb-4aab-9974-09e37b18e6c9","Type":"ContainerDied","Data":"e6338f3a19083748a5a9c9451765ef44141d2b79af29d21d927e4aac562660fd"}
Oct 31 03:48:53.992535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.992417    4900 scope.go:122] "RemoveContainer" containerID="c954017a5e176be6944ea4718890da619129645c57257b42eca514fa5458bdcc"
Oct 31 03:48:53.994387 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.994343    4900 generic.go:356] "Generic (PLEG): container finished" podID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerID="ba93d0fd0f3edcfc90bc1e8d597e76e466d068f3041b752c6bf307d90018cfd9" exitCode=0
Oct 31 03:48:53.995146 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.994836    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4" event={"ID":"a63cf118-4a57-48ba-852b-bcaa10da6d4b","Type":"ContainerDied","Data":"ba93d0fd0f3edcfc90bc1e8d597e76e466d068f3041b752c6bf307d90018cfd9"}
Oct 31 03:48:53.995146 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.994892    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4" event={"ID":"a63cf118-4a57-48ba-852b-bcaa10da6d4b","Type":"ContainerDied","Data":"2305ae837a87b35e0c32a7ce4ddfd199933b1b7cb444c4bd9f017437f0250bb8"}
Oct 31 03:48:53.995146 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:53.994981    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"
Oct 31 03:48:54.004527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.004502    4900 scope.go:122] "RemoveContainer" containerID="c954017a5e176be6944ea4718890da619129645c57257b42eca514fa5458bdcc"
Oct 31 03:48:54.004960 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:48:54.004931    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c954017a5e176be6944ea4718890da619129645c57257b42eca514fa5458bdcc\": container with ID starting with c954017a5e176be6944ea4718890da619129645c57257b42eca514fa5458bdcc not found: ID does not exist" containerID="c954017a5e176be6944ea4718890da619129645c57257b42eca514fa5458bdcc"
Oct 31 03:48:54.005059 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.004965    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c954017a5e176be6944ea4718890da619129645c57257b42eca514fa5458bdcc"} err="failed to get container status \"c954017a5e176be6944ea4718890da619129645c57257b42eca514fa5458bdcc\": rpc error: code = NotFound desc = could not find container \"c954017a5e176be6944ea4718890da619129645c57257b42eca514fa5458bdcc\": container with ID starting with c954017a5e176be6944ea4718890da619129645c57257b42eca514fa5458bdcc not found: ID does not exist"
Oct 31 03:48:54.005059 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.004983    4900 scope.go:122] "RemoveContainer" containerID="ba93d0fd0f3edcfc90bc1e8d597e76e466d068f3041b752c6bf307d90018cfd9"
Oct 31 03:48:54.018341 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.018315    4900 scope.go:122] "RemoveContainer" containerID="97b27b953a11ae70ed57bd26c14b0f15067f80ddcf8682cc28289d0dff2e54ae"
Oct 31 03:48:54.019289 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.019264    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:54.019289 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.019288    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:54.019482 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.019303    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:54.019482 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.019316    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a63cf118-4a57-48ba-852b-bcaa10da6d4b-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:54.019482 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.019330    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:54.019482 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.019344    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhn96\" (UniqueName: \"kubernetes.io/projected/a63cf118-4a57-48ba-852b-bcaa10da6d4b-kube-api-access-rhn96\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:54.019482 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.019358    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pj22\" (UniqueName: \"kubernetes.io/projected/dcc714a7-95eb-4aab-9974-09e37b18e6c9-kube-api-access-9pj22\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:54.019482 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.019372    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:54.019482 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.019384    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcc714a7-95eb-4aab-9974-09e37b18e6c9-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:54.028018 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.027995    4900 scope.go:122] "RemoveContainer" containerID="ba93d0fd0f3edcfc90bc1e8d597e76e466d068f3041b752c6bf307d90018cfd9"
Oct 31 03:48:54.028366 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:48:54.028341    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba93d0fd0f3edcfc90bc1e8d597e76e466d068f3041b752c6bf307d90018cfd9\": container with ID starting with ba93d0fd0f3edcfc90bc1e8d597e76e466d068f3041b752c6bf307d90018cfd9 not found: ID does not exist" containerID="ba93d0fd0f3edcfc90bc1e8d597e76e466d068f3041b752c6bf307d90018cfd9"
Oct 31 03:48:54.028464 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.028374    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba93d0fd0f3edcfc90bc1e8d597e76e466d068f3041b752c6bf307d90018cfd9"} err="failed to get container status \"ba93d0fd0f3edcfc90bc1e8d597e76e466d068f3041b752c6bf307d90018cfd9\": rpc error: code = NotFound desc = could not find container \"ba93d0fd0f3edcfc90bc1e8d597e76e466d068f3041b752c6bf307d90018cfd9\": container with ID starting with ba93d0fd0f3edcfc90bc1e8d597e76e466d068f3041b752c6bf307d90018cfd9 not found: ID does not exist"
Oct 31 03:48:54.028464 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.028393    4900 scope.go:122] "RemoveContainer" containerID="97b27b953a11ae70ed57bd26c14b0f15067f80ddcf8682cc28289d0dff2e54ae"
Oct 31 03:48:54.028704 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:48:54.028682    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97b27b953a11ae70ed57bd26c14b0f15067f80ddcf8682cc28289d0dff2e54ae\": container with ID starting with 97b27b953a11ae70ed57bd26c14b0f15067f80ddcf8682cc28289d0dff2e54ae not found: ID does not exist" containerID="97b27b953a11ae70ed57bd26c14b0f15067f80ddcf8682cc28289d0dff2e54ae"
Oct 31 03:48:54.028749 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.028707    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97b27b953a11ae70ed57bd26c14b0f15067f80ddcf8682cc28289d0dff2e54ae"} err="failed to get container status \"97b27b953a11ae70ed57bd26c14b0f15067f80ddcf8682cc28289d0dff2e54ae\": rpc error: code = NotFound desc = could not find container \"97b27b953a11ae70ed57bd26c14b0f15067f80ddcf8682cc28289d0dff2e54ae\": container with ID starting with 97b27b953a11ae70ed57bd26c14b0f15067f80ddcf8682cc28289d0dff2e54ae not found: ID does not exist"
Oct 31 03:48:54.280098 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.280016    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"]
Oct 31 03:48:54.284319 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.284292    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-phq4n"]
Oct 31 03:48:54.393256 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.393213    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"]
Oct 31 03:48:54.395271 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:54.395250    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-qfkj4"]
Oct 31 03:48:55.152306 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.152272    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" path="/var/lib/kubelet/pods/a63cf118-4a57-48ba-852b-bcaa10da6d4b/volumes"
Oct 31 03:48:55.152798 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.152773    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc714a7-95eb-4aab-9974-09e37b18e6c9" path="/var/lib/kubelet/pods/dcc714a7-95eb-4aab-9974-09e37b18e6c9/volumes"
Oct 31 03:48:55.597620 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.597582    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"]
Oct 31 03:48:55.597842 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.597812    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv" podUID="4dd6e56b-0a06-4a47-abea-12b07f41c2db" containerName="discovery" containerID="cri-o://4a3cfe09600bf1e823e3409a6079815198d9a2ea1ecb8cb59600a5ccf4d32376" gracePeriod=30
Oct 31 03:48:55.647644 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.647608    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"]
Oct 31 03:48:55.648276 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648254    4900 cpu_manager.go:414] "containerMap: removing container" podUID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerName="istio-proxy"
Oct 31 03:48:55.648276 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648274    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerName="istio-proxy"
Oct 31 03:48:55.648548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648287    4900 cpu_manager.go:414] "containerMap: removing container" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerName="istio-proxy"
Oct 31 03:48:55.648548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648292    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerName="istio-proxy"
Oct 31 03:48:55.648548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648300    4900 cpu_manager.go:414] "containerMap: removing container" podUID="dcc714a7-95eb-4aab-9974-09e37b18e6c9" containerName="istio-proxy"
Oct 31 03:48:55.648548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648305    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc714a7-95eb-4aab-9974-09e37b18e6c9" containerName="istio-proxy"
Oct 31 03:48:55.648548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648317    4900 cpu_manager.go:414] "containerMap: removing container" podUID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerName="istio-proxy"
Oct 31 03:48:55.648548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648321    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerName="istio-proxy"
Oct 31 03:48:55.648548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648333    4900 cpu_manager.go:414] "containerMap: removing container" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerName="istio-proxy"
Oct 31 03:48:55.648548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648341    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerName="istio-proxy"
Oct 31 03:48:55.648548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648360    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c72b34a0-8246-4c13-a3a3-49501c50d6e5" containerName="payload-pre-processing"
Oct 31 03:48:55.648548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648364    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72b34a0-8246-4c13-a3a3-49501c50d6e5" containerName="payload-pre-processing"
Oct 31 03:48:55.648548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648470    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc714a7-95eb-4aab-9974-09e37b18e6c9" containerName="istio-proxy"
Oct 31 03:48:55.648548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648482    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerName="istio-proxy"
Oct 31 03:48:55.648548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648493    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerName="istio-proxy"
Oct 31 03:48:55.648548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.648502    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72b34a0-8246-4c13-a3a3-49501c50d6e5" containerName="payload-pre-processing"
Oct 31 03:48:55.651913 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.651891    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.702310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.702273    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"]
Oct 31 03:48:55.740379 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.740336    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.740593 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.740396    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.740593 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.740531    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.740723 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.740631    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.740723 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.740701    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkbl9\" (UniqueName: \"kubernetes.io/projected/6ee9cc43-db19-4ef8-9a17-c576d409d498-kube-api-access-qkbl9\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.740859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.740764    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.740914 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.740868    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.740977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.740911    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.741036 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.741015    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee9cc43-db19-4ef8-9a17-c576d409d498-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.842304 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.842269    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkbl9\" (UniqueName: \"kubernetes.io/projected/6ee9cc43-db19-4ef8-9a17-c576d409d498-kube-api-access-qkbl9\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.842497 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.842340    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.842497 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.842407    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.842497 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.842440    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.842497 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.842483    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee9cc43-db19-4ef8-9a17-c576d409d498-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.842719 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.842522    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.842719 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.842552    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.842719 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.842630    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.842719 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.842703    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.842975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.842948    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.843037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.842976    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.843205 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.843167    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.843334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.843220    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.843439 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.843414    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee9cc43-db19-4ef8-9a17-c576d409d498-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.845023 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.844999    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.845234 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.845216    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.875316 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.875286    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:48:55.901590 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.901555    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.901744 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.901704    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkbl9\" (UniqueName: \"kubernetes.io/projected/6ee9cc43-db19-4ef8-9a17-c576d409d498-kube-api-access-qkbl9\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:55.943419 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.943381    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-token\" (UniqueName: \"kubernetes.io/projected/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-token\") pod \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") "
Oct 31 03:48:55.943419 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.943433    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-csr-ca-configmap\") pod \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") "
Oct 31 03:48:55.943686 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.943496    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dd6e56b-0a06-4a47-abea-12b07f41c2db-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd6e56b-0a06-4a47-abea-12b07f41c2db-local-certs\") pod \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") "
Oct 31 03:48:55.943686 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.943524    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-kubeconfig\") pod \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") "
Oct 31 03:48:55.943686 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.943576    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd6e56b-0a06-4a47-abea-12b07f41c2db-kube-api-access-8pztj\" (UniqueName: \"kubernetes.io/projected/4dd6e56b-0a06-4a47-abea-12b07f41c2db-kube-api-access-8pztj\") pod \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") "
Oct 31 03:48:55.943686 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.943607    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-cacerts\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-cacerts\") pod \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") "
Oct 31 03:48:55.943934 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.943727    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-csr-dns-cert\") pod \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\" (UID: \"4dd6e56b-0a06-4a47-abea-12b07f41c2db\") "
Oct 31 03:48:55.944201 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.943972    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-csr-ca-configmap" pod "4dd6e56b-0a06-4a47-abea-12b07f41c2db" (UID: "4dd6e56b-0a06-4a47-abea-12b07f41c2db"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:55.944350 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.944310    4900 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-csr-ca-configmap\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:55.945970 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.945943    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dd6e56b-0a06-4a47-abea-12b07f41c2db-local-certs" pod "4dd6e56b-0a06-4a47-abea-12b07f41c2db" (UID: "4dd6e56b-0a06-4a47-abea-12b07f41c2db"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:55.946085 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.945943    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-cacerts" pod "4dd6e56b-0a06-4a47-abea-12b07f41c2db" (UID: "4dd6e56b-0a06-4a47-abea-12b07f41c2db"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:55.946163 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.946146    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-token" pod "4dd6e56b-0a06-4a47-abea-12b07f41c2db" (UID: "4dd6e56b-0a06-4a47-abea-12b07f41c2db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:55.946265 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.946245    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-kubeconfig" pod "4dd6e56b-0a06-4a47-abea-12b07f41c2db" (UID: "4dd6e56b-0a06-4a47-abea-12b07f41c2db"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:55.946310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.946265    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-csr-dns-cert" pod "4dd6e56b-0a06-4a47-abea-12b07f41c2db" (UID: "4dd6e56b-0a06-4a47-abea-12b07f41c2db"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:55.946310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.946284    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd6e56b-0a06-4a47-abea-12b07f41c2db-kube-api-access-8pztj" pod "4dd6e56b-0a06-4a47-abea-12b07f41c2db" (UID: "4dd6e56b-0a06-4a47-abea-12b07f41c2db"). InnerVolumeSpecName "kube-api-access-8pztj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:55.997274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:55.997240    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:56.008473 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.008438    4900 generic.go:356] "Generic (PLEG): container finished" podID="4dd6e56b-0a06-4a47-abea-12b07f41c2db" containerID="4a3cfe09600bf1e823e3409a6079815198d9a2ea1ecb8cb59600a5ccf4d32376" exitCode=0
Oct 31 03:48:56.008642 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.008492    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"
Oct 31 03:48:56.008642 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.008527    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv" event={"ID":"4dd6e56b-0a06-4a47-abea-12b07f41c2db","Type":"ContainerDied","Data":"4a3cfe09600bf1e823e3409a6079815198d9a2ea1ecb8cb59600a5ccf4d32376"}
Oct 31 03:48:56.008642 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.008577    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv" event={"ID":"4dd6e56b-0a06-4a47-abea-12b07f41c2db","Type":"ContainerDied","Data":"a17092633cc26d84f70eb74e7e0ac2553a958c793f137a0ed6c46c00a806dff1"}
Oct 31 03:48:56.008642 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.008602    4900 scope.go:122] "RemoveContainer" containerID="4a3cfe09600bf1e823e3409a6079815198d9a2ea1ecb8cb59600a5ccf4d32376"
Oct 31 03:48:56.020548 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.020521    4900 scope.go:122] "RemoveContainer" containerID="4a3cfe09600bf1e823e3409a6079815198d9a2ea1ecb8cb59600a5ccf4d32376"
Oct 31 03:48:56.020947 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:48:56.020909    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3cfe09600bf1e823e3409a6079815198d9a2ea1ecb8cb59600a5ccf4d32376\": container with ID starting with 4a3cfe09600bf1e823e3409a6079815198d9a2ea1ecb8cb59600a5ccf4d32376 not found: ID does not exist" containerID="4a3cfe09600bf1e823e3409a6079815198d9a2ea1ecb8cb59600a5ccf4d32376"
Oct 31 03:48:56.021032 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.020948    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3cfe09600bf1e823e3409a6079815198d9a2ea1ecb8cb59600a5ccf4d32376"} err="failed to get container status \"4a3cfe09600bf1e823e3409a6079815198d9a2ea1ecb8cb59600a5ccf4d32376\": rpc error: code = NotFound desc = could not find container \"4a3cfe09600bf1e823e3409a6079815198d9a2ea1ecb8cb59600a5ccf4d32376\": container with ID starting with 4a3cfe09600bf1e823e3409a6079815198d9a2ea1ecb8cb59600a5ccf4d32376 not found: ID does not exist"
Oct 31 03:48:56.046173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.046134    4900 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-kubeconfig\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:56.046173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.046172    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pztj\" (UniqueName: \"kubernetes.io/projected/4dd6e56b-0a06-4a47-abea-12b07f41c2db-kube-api-access-8pztj\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:56.046434 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.046190    4900 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-cacerts\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:56.046434 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.046205    4900 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-csr-dns-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:56.046434 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.046219    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dd6e56b-0a06-4a47-abea-12b07f41c2db-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:56.046434 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.046230    4900 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dd6e56b-0a06-4a47-abea-12b07f41c2db-local-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:56.164087 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.163982    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:56.175349 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.175316    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"]
Oct 31 03:48:56.180951 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.180923    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrklv"]
Oct 31 03:48:56.373314 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.373279    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:56.420720 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:48:56.420679    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee9cc43_db19_4ef8_9a17_c576d409d498.slice/crio-b5e053150794e20f15bac7602d4a83e25c5817cf58e40d3eb0f4a4ef7e3109ab WatchSource:0}: Error finding container b5e053150794e20f15bac7602d4a83e25c5817cf58e40d3eb0f4a4ef7e3109ab: Status 404 returned error can't find the container with id b5e053150794e20f15bac7602d4a83e25c5817cf58e40d3eb0f4a4ef7e3109ab
Oct 31 03:48:56.422671 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.422648    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"]
Oct 31 03:48:56.423194 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.423164    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:48:56.423245 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.423226    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:48:56.423288 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:56.423254    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:48:57.016837 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:57.016798    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p" event={"ID":"6ee9cc43-db19-4ef8-9a17-c576d409d498","Type":"ContainerStarted","Data":"20b51f0779bca56dbf1dfca73b761d60b891c7df3541fadad043bfe8f2bb5fb6"}
Oct 31 03:48:57.017047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:57.016857    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p" event={"ID":"6ee9cc43-db19-4ef8-9a17-c576d409d498","Type":"ContainerStarted","Data":"b5e053150794e20f15bac7602d4a83e25c5817cf58e40d3eb0f4a4ef7e3109ab"}
Oct 31 03:48:57.146827 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:57.134734    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p" podStartSLOduration=2.134714748 podStartE2EDuration="2.134714748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:57.133107856 +0000 UTC m=+2490.099441201" watchObservedRunningTime="2026-10-31 03:48:57.134714748 +0000 UTC m=+2490.101048093"
Oct 31 03:48:57.169651 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:57.169605    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd6e56b-0a06-4a47-abea-12b07f41c2db" path="/var/lib/kubelet/pods/4dd6e56b-0a06-4a47-abea-12b07f41c2db/volumes"
Oct 31 03:48:57.997822 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:57.997766    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:48:57.999029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:57.999002    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:48:57.999171 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:57.999051    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p" podUID="6ee9cc43-db19-4ef8-9a17-c576d409d498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 03:48:58.968599 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:58.968561    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"]
Oct 31 03:48:58.969121 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:58.968769    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55" podUID="7c853068-dbe4-42a1-83d8-571b1b2b1719" containerName="istio-proxy" containerID="cri-o://9545efefdde531249cb84e0abda6048220c8f8ff20ee7dff1088f563c408cfd7" gracePeriod=30
Oct 31 03:48:58.997888 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:58.997859    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:48:58.998030 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:58.997910    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p" podUID="6ee9cc43-db19-4ef8-9a17-c576d409d498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 03:48:59.273104 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:59.273068    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:59.482250 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:59.482198    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:48:59.776596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:59.776563    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"]
Oct 31 03:48:59.881142 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:59.881109    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"]
Oct 31 03:48:59.998010 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:59.997977    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:48:59.998472 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:48:59.998033    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p" podUID="6ee9cc43-db19-4ef8-9a17-c576d409d498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 03:49:00.209614 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:00.209530    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:00.418766 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:00.418734    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:00.997865 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:00.997827    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:49:00.998051 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:00.997881    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p" podUID="6ee9cc43-db19-4ef8-9a17-c576d409d498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 03:49:01.997854 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:01.997783    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:49:01.998047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:01.997869    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p" podUID="6ee9cc43-db19-4ef8-9a17-c576d409d498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 03:49:02.998310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:02.998274    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:49:02.998712 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:02.998339    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p" podUID="6ee9cc43-db19-4ef8-9a17-c576d409d498" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 31 03:49:04.002632 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.002604    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:49:04.003017 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.002903    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:49:04.003525 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.003507    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 03:49:04.223246 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.223220    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:49:04.344117 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.344079    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c853068-dbe4-42a1-83d8-571b1b2b1719-kube-api-access-qcxh7\" (UniqueName: \"kubernetes.io/projected/7c853068-dbe4-42a1-83d8-571b1b2b1719-kube-api-access-qcxh7\") pod \"7c853068-dbe4-42a1-83d8-571b1b2b1719\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") "
Oct 31 03:49:04.344299 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.344143    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-credential-socket\") pod \"7c853068-dbe4-42a1-83d8-571b1b2b1719\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") "
Oct 31 03:49:04.344299 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.344205    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-podinfo\") pod \"7c853068-dbe4-42a1-83d8-571b1b2b1719\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") "
Oct 31 03:49:04.344299 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.344251    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c853068-dbe4-42a1-83d8-571b1b2b1719-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c853068-dbe4-42a1-83d8-571b1b2b1719-istiod-ca-cert\") pod \"7c853068-dbe4-42a1-83d8-571b1b2b1719\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") "
Oct 31 03:49:04.344299 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.344297    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-workload-certs\") pod \"7c853068-dbe4-42a1-83d8-571b1b2b1719\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") "
Oct 31 03:49:04.344553 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.344397    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-envoy\") pod \"7c853068-dbe4-42a1-83d8-571b1b2b1719\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") "
Oct 31 03:49:04.344553 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.344444    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-data\") pod \"7c853068-dbe4-42a1-83d8-571b1b2b1719\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") "
Oct 31 03:49:04.344553 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.344478    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-token\" (UniqueName: \"kubernetes.io/projected/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-token\") pod \"7c853068-dbe4-42a1-83d8-571b1b2b1719\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") "
Oct 31 03:49:04.344553 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.344507    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-workload-socket\") pod \"7c853068-dbe4-42a1-83d8-571b1b2b1719\" (UID: \"7c853068-dbe4-42a1-83d8-571b1b2b1719\") "
Oct 31 03:49:04.344553 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.344541    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-credential-socket" pod "7c853068-dbe4-42a1-83d8-571b1b2b1719" (UID: "7c853068-dbe4-42a1-83d8-571b1b2b1719"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:04.344849 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.344684    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c853068-dbe4-42a1-83d8-571b1b2b1719-istiod-ca-cert" pod "7c853068-dbe4-42a1-83d8-571b1b2b1719" (UID: "7c853068-dbe4-42a1-83d8-571b1b2b1719"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:04.344961 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.344935    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-workload-socket" pod "7c853068-dbe4-42a1-83d8-571b1b2b1719" (UID: "7c853068-dbe4-42a1-83d8-571b1b2b1719"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:04.344961 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.344946    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-data" pod "7c853068-dbe4-42a1-83d8-571b1b2b1719" (UID: "7c853068-dbe4-42a1-83d8-571b1b2b1719"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:04.345077 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.344684    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-workload-certs" pod "7c853068-dbe4-42a1-83d8-571b1b2b1719" (UID: "7c853068-dbe4-42a1-83d8-571b1b2b1719"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:04.345255 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.345231    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:04.345255 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.345255    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:04.345444 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.345265    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:04.345444 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.345274    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:04.345444 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.345283    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c853068-dbe4-42a1-83d8-571b1b2b1719-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:04.346492 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.346467    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c853068-dbe4-42a1-83d8-571b1b2b1719-kube-api-access-qcxh7" pod "7c853068-dbe4-42a1-83d8-571b1b2b1719" (UID: "7c853068-dbe4-42a1-83d8-571b1b2b1719"). InnerVolumeSpecName "kube-api-access-qcxh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:04.346594 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.346572    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-podinfo" pod "7c853068-dbe4-42a1-83d8-571b1b2b1719" (UID: "7c853068-dbe4-42a1-83d8-571b1b2b1719"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:49:04.346707 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.346689    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-token" pod "7c853068-dbe4-42a1-83d8-571b1b2b1719" (UID: "7c853068-dbe4-42a1-83d8-571b1b2b1719"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:04.347089 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.347071    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-envoy" pod "7c853068-dbe4-42a1-83d8-571b1b2b1719" (UID: "7c853068-dbe4-42a1-83d8-571b1b2b1719"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:49:04.446905 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.446871    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:04.446905 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.446901    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:04.446905 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.446910    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c853068-dbe4-42a1-83d8-571b1b2b1719-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:04.447143 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.446920    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcxh7\" (UniqueName: \"kubernetes.io/projected/7c853068-dbe4-42a1-83d8-571b1b2b1719-kube-api-access-qcxh7\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:04.492211 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.492181    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:04.777056 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.777010    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2" podUID="3942dcff-b885-43c1-bee5-bfca753b1df0" containerName="payload-processing" containerID="cri-o://6d0d30e9a6de1964d5ff9491390114ff9eadfae0f6426472b5131387fd8721df" gracePeriod=25
Oct 31 03:49:04.881760 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:04.881702    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp" podUID="60ded06c-f133-49af-9ed0-7dc31f49e95c" containerName="payload-pre-processing" containerID="cri-o://0748210bf7f2e751820bf84407347b7a9bdb7eb27a12006a8eee18d67f127453" gracePeriod=25
Oct 31 03:49:05.040296 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.040228    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"
Oct 31 03:49:05.045983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.045957    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:05.092800 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.092766    4900 generic.go:356] "Generic (PLEG): container finished" podID="3942dcff-b885-43c1-bee5-bfca753b1df0" containerID="6d0d30e9a6de1964d5ff9491390114ff9eadfae0f6426472b5131387fd8721df" exitCode=0
Oct 31 03:49:05.092965 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.092916    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"
Oct 31 03:49:05.093072 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.093057    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2" event={"ID":"3942dcff-b885-43c1-bee5-bfca753b1df0","Type":"ContainerDied","Data":"6d0d30e9a6de1964d5ff9491390114ff9eadfae0f6426472b5131387fd8721df"}
Oct 31 03:49:05.093144 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.093092    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2" event={"ID":"3942dcff-b885-43c1-bee5-bfca753b1df0","Type":"ContainerDied","Data":"fcac57cbdf97cca8f0a35a091e6aa3dabc00086da40878b0dd393199d247fb57"}
Oct 31 03:49:05.093144 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.093113    4900 scope.go:122] "RemoveContainer" containerID="6d0d30e9a6de1964d5ff9491390114ff9eadfae0f6426472b5131387fd8721df"
Oct 31 03:49:05.094767 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.094742    4900 generic.go:356] "Generic (PLEG): container finished" podID="7c853068-dbe4-42a1-83d8-571b1b2b1719" containerID="9545efefdde531249cb84e0abda6048220c8f8ff20ee7dff1088f563c408cfd7" exitCode=0
Oct 31 03:49:05.095403 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.094934    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55" event={"ID":"7c853068-dbe4-42a1-83d8-571b1b2b1719","Type":"ContainerDied","Data":"9545efefdde531249cb84e0abda6048220c8f8ff20ee7dff1088f563c408cfd7"}
Oct 31 03:49:05.095403 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.094962    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"
Oct 31 03:49:05.095403 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.094991    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55" event={"ID":"7c853068-dbe4-42a1-83d8-571b1b2b1719","Type":"ContainerDied","Data":"a324af98f5378fe1d4b108243e2054ad11f5f8a0a8542fe0710ec6c1dae8c7f1"}
Oct 31 03:49:05.103109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.102073    4900 generic.go:356] "Generic (PLEG): container finished" podID="60ded06c-f133-49af-9ed0-7dc31f49e95c" containerID="0748210bf7f2e751820bf84407347b7a9bdb7eb27a12006a8eee18d67f127453" exitCode=0
Oct 31 03:49:05.103109 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.102879    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp" event={"ID":"60ded06c-f133-49af-9ed0-7dc31f49e95c","Type":"ContainerDied","Data":"0748210bf7f2e751820bf84407347b7a9bdb7eb27a12006a8eee18d67f127453"}
Oct 31 03:49:05.104831 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.104780    4900 scope.go:122] "RemoveContainer" containerID="6d0d30e9a6de1964d5ff9491390114ff9eadfae0f6426472b5131387fd8721df"
Oct 31 03:49:05.105285 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:49:05.105132    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0d30e9a6de1964d5ff9491390114ff9eadfae0f6426472b5131387fd8721df\": container with ID starting with 6d0d30e9a6de1964d5ff9491390114ff9eadfae0f6426472b5131387fd8721df not found: ID does not exist" containerID="6d0d30e9a6de1964d5ff9491390114ff9eadfae0f6426472b5131387fd8721df"
Oct 31 03:49:05.105285 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.105169    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0d30e9a6de1964d5ff9491390114ff9eadfae0f6426472b5131387fd8721df"} err="failed to get container status \"6d0d30e9a6de1964d5ff9491390114ff9eadfae0f6426472b5131387fd8721df\": rpc error: code = NotFound desc = could not find container \"6d0d30e9a6de1964d5ff9491390114ff9eadfae0f6426472b5131387fd8721df\": container with ID starting with 6d0d30e9a6de1964d5ff9491390114ff9eadfae0f6426472b5131387fd8721df not found: ID does not exist"
Oct 31 03:49:05.105285 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.105192    4900 scope.go:122] "RemoveContainer" containerID="9545efefdde531249cb84e0abda6048220c8f8ff20ee7dff1088f563c408cfd7"
Oct 31 03:49:05.115367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.115342    4900 scope.go:122] "RemoveContainer" containerID="9545efefdde531249cb84e0abda6048220c8f8ff20ee7dff1088f563c408cfd7"
Oct 31 03:49:05.115651 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:49:05.115630    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9545efefdde531249cb84e0abda6048220c8f8ff20ee7dff1088f563c408cfd7\": container with ID starting with 9545efefdde531249cb84e0abda6048220c8f8ff20ee7dff1088f563c408cfd7 not found: ID does not exist" containerID="9545efefdde531249cb84e0abda6048220c8f8ff20ee7dff1088f563c408cfd7"
Oct 31 03:49:05.115718 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.115657    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9545efefdde531249cb84e0abda6048220c8f8ff20ee7dff1088f563c408cfd7"} err="failed to get container status \"9545efefdde531249cb84e0abda6048220c8f8ff20ee7dff1088f563c408cfd7\": rpc error: code = NotFound desc = could not find container \"9545efefdde531249cb84e0abda6048220c8f8ff20ee7dff1088f563c408cfd7\": container with ID starting with 9545efefdde531249cb84e0abda6048220c8f8ff20ee7dff1088f563c408cfd7 not found: ID does not exist"
Oct 31 03:49:05.154646 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.154398    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3942dcff-b885-43c1-bee5-bfca753b1df0-kube-api-access-tdshr\" (UniqueName: \"kubernetes.io/projected/3942dcff-b885-43c1-bee5-bfca753b1df0-kube-api-access-tdshr\") pod \"3942dcff-b885-43c1-bee5-bfca753b1df0\" (UID: \"3942dcff-b885-43c1-bee5-bfca753b1df0\") "
Oct 31 03:49:05.154646 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.154535    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3942dcff-b885-43c1-bee5-bfca753b1df0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3942dcff-b885-43c1-bee5-bfca753b1df0-plugins-config-volume\") pod \"3942dcff-b885-43c1-bee5-bfca753b1df0\" (UID: \"3942dcff-b885-43c1-bee5-bfca753b1df0\") "
Oct 31 03:49:05.158825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.155038    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"
Oct 31 03:49:05.158825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.156245    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3942dcff-b885-43c1-bee5-bfca753b1df0-plugins-config-volume" pod "3942dcff-b885-43c1-bee5-bfca753b1df0" (UID: "3942dcff-b885-43c1-bee5-bfca753b1df0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:05.158825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.157663    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3942dcff-b885-43c1-bee5-bfca753b1df0-kube-api-access-tdshr" pod "3942dcff-b885-43c1-bee5-bfca753b1df0" (UID: "3942dcff-b885-43c1-bee5-bfca753b1df0"). InnerVolumeSpecName "kube-api-access-tdshr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:05.255524 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.255489    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ded06c-f133-49af-9ed0-7dc31f49e95c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ded06c-f133-49af-9ed0-7dc31f49e95c-plugins-config-volume\") pod \"60ded06c-f133-49af-9ed0-7dc31f49e95c\" (UID: \"60ded06c-f133-49af-9ed0-7dc31f49e95c\") "
Oct 31 03:49:05.255698 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.255632    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ded06c-f133-49af-9ed0-7dc31f49e95c-kube-api-access-bnfpz\" (UniqueName: \"kubernetes.io/projected/60ded06c-f133-49af-9ed0-7dc31f49e95c-kube-api-access-bnfpz\") pod \"60ded06c-f133-49af-9ed0-7dc31f49e95c\" (UID: \"60ded06c-f133-49af-9ed0-7dc31f49e95c\") "
Oct 31 03:49:05.255952 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.255920    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ded06c-f133-49af-9ed0-7dc31f49e95c-plugins-config-volume" pod "60ded06c-f133-49af-9ed0-7dc31f49e95c" (UID: "60ded06c-f133-49af-9ed0-7dc31f49e95c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:49:05.255952 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.255944    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdshr\" (UniqueName: \"kubernetes.io/projected/3942dcff-b885-43c1-bee5-bfca753b1df0-kube-api-access-tdshr\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:05.255952 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.255956    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3942dcff-b885-43c1-bee5-bfca753b1df0-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:05.257636 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.257618    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ded06c-f133-49af-9ed0-7dc31f49e95c-kube-api-access-bnfpz" pod "60ded06c-f133-49af-9ed0-7dc31f49e95c" (UID: "60ded06c-f133-49af-9ed0-7dc31f49e95c"). InnerVolumeSpecName "kube-api-access-bnfpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:49:05.356747 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.356654    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnfpz\" (UniqueName: \"kubernetes.io/projected/60ded06c-f133-49af-9ed0-7dc31f49e95c-kube-api-access-bnfpz\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:05.356747 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.356690    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ded06c-f133-49af-9ed0-7dc31f49e95c-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:49:05.380091 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.380060    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"]
Oct 31 03:49:05.429723 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.429684    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-sjq55"]
Oct 31 03:49:05.561753 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.561704    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"]
Oct 31 03:49:05.571721 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:05.571688    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-fc85c7-5bd95745d5-kfsj2"]
Oct 31 03:49:06.109489 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:06.109454    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"
Oct 31 03:49:06.109489 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:06.109469    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp" event={"ID":"60ded06c-f133-49af-9ed0-7dc31f49e95c","Type":"ContainerDied","Data":"81b6668226d84fb53109bd36b4ff0cf8df4e9e90ab4290b370bef2a1857d25ac"}
Oct 31 03:49:06.110080 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:06.109662    4900 scope.go:122] "RemoveContainer" containerID="0748210bf7f2e751820bf84407347b7a9bdb7eb27a12006a8eee18d67f127453"
Oct 31 03:49:06.275862 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:06.275113    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"]
Oct 31 03:49:06.281770 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:06.281727    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fc85c7-7996b7c78c-5jwhp"]
Oct 31 03:49:07.005275 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:49:07.005134    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:49:07.154839 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:07.154804    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3942dcff-b885-43c1-bee5-bfca753b1df0" path="/var/lib/kubelet/pods/3942dcff-b885-43c1-bee5-bfca753b1df0/volumes"
Oct 31 03:49:07.155414 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:07.155391    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ded06c-f133-49af-9ed0-7dc31f49e95c" path="/var/lib/kubelet/pods/60ded06c-f133-49af-9ed0-7dc31f49e95c/volumes"
Oct 31 03:49:07.156110 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:07.156088    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c853068-dbe4-42a1-83d8-571b1b2b1719" path="/var/lib/kubelet/pods/7c853068-dbe4-42a1-83d8-571b1b2b1719/volumes"
Oct 31 03:49:09.702449 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:09.702415    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:11.540643 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:11.540605    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:11.748594 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:11.748555    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:14.581378 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:14.581337    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:19.543080 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:19.543048    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:23.714146 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:23.714114    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:23.923466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:23.923431    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:24.827479 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:24.827438    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:29.215872 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.215836    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"]
Oct 31 03:49:29.216283 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.216270    4900 cpu_manager.go:414] "containerMap: removing container" podUID="4dd6e56b-0a06-4a47-abea-12b07f41c2db" containerName="discovery"
Oct 31 03:49:29.216327 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.216283    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd6e56b-0a06-4a47-abea-12b07f41c2db" containerName="discovery"
Oct 31 03:49:29.216327 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.216309    4900 cpu_manager.go:414] "containerMap: removing container" podUID="60ded06c-f133-49af-9ed0-7dc31f49e95c" containerName="payload-pre-processing"
Oct 31 03:49:29.216327 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.216314    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ded06c-f133-49af-9ed0-7dc31f49e95c" containerName="payload-pre-processing"
Oct 31 03:49:29.216431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.216328    4900 cpu_manager.go:414] "containerMap: removing container" podUID="7c853068-dbe4-42a1-83d8-571b1b2b1719" containerName="istio-proxy"
Oct 31 03:49:29.216431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.216333    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c853068-dbe4-42a1-83d8-571b1b2b1719" containerName="istio-proxy"
Oct 31 03:49:29.216431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.216351    4900 cpu_manager.go:414] "containerMap: removing container" podUID="3942dcff-b885-43c1-bee5-bfca753b1df0" containerName="payload-processing"
Oct 31 03:49:29.216431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.216356    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3942dcff-b885-43c1-bee5-bfca753b1df0" containerName="payload-processing"
Oct 31 03:49:29.216431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.216424    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="a63cf118-4a57-48ba-852b-bcaa10da6d4b" containerName="istio-proxy"
Oct 31 03:49:29.216578 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.216433    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f07fc2e-b24b-46c6-9571-39282e304a53" containerName="istio-proxy"
Oct 31 03:49:29.216578 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.216441    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ded06c-f133-49af-9ed0-7dc31f49e95c" containerName="payload-pre-processing"
Oct 31 03:49:29.216578 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.216449    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c853068-dbe4-42a1-83d8-571b1b2b1719" containerName="istio-proxy"
Oct 31 03:49:29.216578 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.216457    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="3942dcff-b885-43c1-bee5-bfca753b1df0" containerName="payload-processing"
Oct 31 03:49:29.216578 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.216464    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd6e56b-0a06-4a47-abea-12b07f41c2db" containerName="discovery"
Oct 31 03:49:29.220934 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.220918    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.275160 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.275125    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"]
Oct 31 03:49:29.347457 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.347425    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.347633 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.347472    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.347633 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.347590    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.347715 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.347650    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.347751 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.347724    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.347804 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.347771    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.347843 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.347809    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6dvb\" (UniqueName: \"kubernetes.io/projected/7092ff60-28f3-4737-897c-41020bc4a5a7-kube-api-access-p6dvb\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.347897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.347882    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7092ff60-28f3-4737-897c-41020bc4a5a7-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.347942 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.347911    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.448520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.448488    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.448705 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.448549    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.448705 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.448577    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.448705 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.448611    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.448705 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.448629    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6dvb\" (UniqueName: \"kubernetes.io/projected/7092ff60-28f3-4737-897c-41020bc4a5a7-kube-api-access-p6dvb\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.448705 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.448674    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7092ff60-28f3-4737-897c-41020bc4a5a7-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.449003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.448714    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.449003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.448740    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.449003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.448784    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.449003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.448969    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.449243 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.449015    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.449303 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.449244    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.449303 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.449284    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.449576 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.449556    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7092ff60-28f3-4737-897c-41020bc4a5a7-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.451048 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.451025    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.451129 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.451077    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.505499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.505464    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.505668 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.505649    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6dvb\" (UniqueName: \"kubernetes.io/projected/7092ff60-28f3-4737-897c-41020bc4a5a7-kube-api-access-p6dvb\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.535708 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.535678    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:29.951626 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:49:29.951586    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7092ff60_28f3_4737_897c_41020bc4a5a7.slice/crio-32d503ade1f24838a60817626fd89777fa14cf0881354258cac2fb14d6dc9b3b WatchSource:0}: Error finding container 32d503ade1f24838a60817626fd89777fa14cf0881354258cac2fb14d6dc9b3b: Status 404 returned error can't find the container with id 32d503ade1f24838a60817626fd89777fa14cf0881354258cac2fb14d6dc9b3b
Oct 31 03:49:29.953670 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.953650    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"]
Oct 31 03:49:29.954188 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.954156    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:49:29.954259 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.954218    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:49:29.954259 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:29.954246    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:49:30.072001 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:30.071970    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:30.257164 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:30.257122    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl" event={"ID":"7092ff60-28f3-4737-897c-41020bc4a5a7","Type":"ContainerStarted","Data":"d4e19d0f8df7d8859832cbc3feaa55fc301c098976db8f75e663ca003b68e24a"}
Oct 31 03:49:30.257627 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:30.257183    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl" event={"ID":"7092ff60-28f3-4737-897c-41020bc4a5a7","Type":"ContainerStarted","Data":"32d503ade1f24838a60817626fd89777fa14cf0881354258cac2fb14d6dc9b3b"}
Oct 31 03:49:30.375027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:30.374974    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl" podStartSLOduration=1.374959931 podStartE2EDuration="1.374959931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:30.373275227 +0000 UTC m=+2523.339608582" watchObservedRunningTime="2026-10-31 03:49:30.374959931 +0000 UTC m=+2523.341293274"
Oct 31 03:49:30.554875 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:30.553885    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:31.538814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:31.537779    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:49:31.538814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:31.537885    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl" podUID="7092ff60-28f3-4737-897c-41020bc4a5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 03:49:32.538815 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:32.538230    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:49:32.538815 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:32.538288    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl" podUID="7092ff60-28f3-4737-897c-41020bc4a5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 03:49:33.547814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:33.538223    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:49:33.547814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:33.538290    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl" podUID="7092ff60-28f3-4737-897c-41020bc4a5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 03:49:34.557298 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:34.557256    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:49:34.557974 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:34.557941    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl" podUID="7092ff60-28f3-4737-897c-41020bc4a5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 03:49:35.173179 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:35.173140    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:35.545813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:35.539215    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:49:35.545813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:35.539281    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl" podUID="7092ff60-28f3-4737-897c-41020bc4a5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 03:49:36.536215 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:36.536177    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 03:49:36.536622 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:36.536233    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl" podUID="7092ff60-28f3-4737-897c-41020bc4a5a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 03:49:36.990758 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:49:36.990715    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:49:38.353880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:38.353841    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:38.354309 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:38.354206    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:38.354881 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:38.354853    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:49:39.450601 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:39.450529    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:40.983848 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:40.983815    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"]
Oct 31 03:49:40.987580 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:40.987563    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.031537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.031480    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.031694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.031584    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.031694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.031609    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.031694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.031689    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.031862 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.031707    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.031862 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.031725    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s87r\" (UniqueName: \"kubernetes.io/projected/cf89487b-5877-4f0e-81f7-abb1ff44d51a-kube-api-access-7s87r\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.031862 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.031763    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.031960 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.031880    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.031960 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.031899    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.097797 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.097748    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"]
Oct 31 03:49:41.133233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.133202    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.133427 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.133259    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.133427 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.133292    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.133427 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.133338    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.133427 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.133362    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.133675 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.133464    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s87r\" (UniqueName: \"kubernetes.io/projected/cf89487b-5877-4f0e-81f7-abb1ff44d51a-kube-api-access-7s87r\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.133675 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.133525    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.133675 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.133596    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.133675 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.133625    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.133894 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.133694    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-data\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.133894 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.133719    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-workload-certs\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.134131 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.134108    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istiod-ca-cert\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.134131 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.134122    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-workload-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.134314 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.134178    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-credential-socket\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.135883 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.135856    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-envoy\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.136003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.135985    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-podinfo\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.190218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.190183    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-token\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.190399 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.190376    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s87r\" (UniqueName: \"kubernetes.io/projected/cf89487b-5877-4f0e-81f7-abb1ff44d51a-kube-api-access-7s87r\") pod \"e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") " pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.301031 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.300949    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:41.584442 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.584366    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:41.717243 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:49:41.717213    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf89487b_5877_4f0e_81f7_abb1ff44d51a.slice/crio-82c28a71be47235ac643e8a4c3426b2ee45b11128127aa35ceb7c0607a5dd03b WatchSource:0}: Error finding container 82c28a71be47235ac643e8a4c3426b2ee45b11128127aa35ceb7c0607a5dd03b: Status 404 returned error can't find the container with id 82c28a71be47235ac643e8a4c3426b2ee45b11128127aa35ceb7c0607a5dd03b
Oct 31 03:49:41.719396 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.719374    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"]
Oct 31 03:49:41.719712 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.719682    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:49:41.719756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.719741    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:49:41.719824 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:41.719770    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:49:42.365612 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:42.365578    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb" event={"ID":"cf89487b-5877-4f0e-81f7-abb1ff44d51a","Type":"ContainerStarted","Data":"36b6df5dcd052d9e9b6f4af182cce74e3addf572b80b9de8d437a7e9e3f14572"}
Oct 31 03:49:42.366123 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:42.365634    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb" event={"ID":"cf89487b-5877-4f0e-81f7-abb1ff44d51a","Type":"ContainerStarted","Data":"82c28a71be47235ac643e8a4c3426b2ee45b11128127aa35ceb7c0607a5dd03b"}
Oct 31 03:49:42.598240 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:42.598191    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb" podStartSLOduration=2.598177764 podStartE2EDuration="2.598177764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:42.554384685 +0000 UTC m=+2535.520718075" watchObservedRunningTime="2026-10-31 03:49:42.598177764 +0000 UTC m=+2535.564511102"
Oct 31 03:49:43.301194 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:43.301156    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:43.302874 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:43.302847    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:49:43.303012 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:43.302900    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb" podUID="cf89487b-5877-4f0e-81f7-abb1ff44d51a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 03:49:44.301687 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:44.301650    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:49:44.302122 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:44.301703    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb" podUID="cf89487b-5877-4f0e-81f7-abb1ff44d51a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 03:49:44.548250 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:44.548216    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:45.301451 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:45.301412    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:49:45.301638 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:45.301468    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb" podUID="cf89487b-5877-4f0e-81f7-abb1ff44d51a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 03:49:45.655117 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:45.655025    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:45.873469 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:45.873438    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:46.206078 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.206045    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"]
Oct 31 03:49:46.210127 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.210107    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.310058 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.310017    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-e2f0bd-openshift-default-dockercfg-kckzx\""
Oct 31 03:49:46.313714 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.310701    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:49:46.314062 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.314011    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb" podUID="cf89487b-5877-4f0e-81f7-abb1ff44d51a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 03:49:46.316623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.316601    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"]
Oct 31 03:49:46.372633 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.372589    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:46.410611 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.410572    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-token\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.410796 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.410704    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-credential-socket\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.410878 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.410819    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-data\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.410926 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.410893    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-workload-socket\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.410971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.410945    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-podinfo\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.411024 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.410985    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbhcp\" (UniqueName: \"kubernetes.io/projected/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-kube-api-access-mbhcp\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.411116 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.411088    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-workload-certs\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.411270 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.411198    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istiod-ca-cert\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.411270 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.411241    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-envoy\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.512302 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.512269    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-envoy\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.512492 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.512361    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-token\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.512492 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.512393    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-credential-socket\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.512492 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.512437    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-data\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.512671 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.512636    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-workload-socket\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.512838 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.512815    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-data\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.512924 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.512868    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-credential-socket\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.512989 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.512968    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-workload-socket\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.513137 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.513114    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-podinfo\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.513200 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.513185    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbhcp\" (UniqueName: \"kubernetes.io/projected/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-kube-api-access-mbhcp\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.513254 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.513236    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-workload-certs\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.513348 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.513327    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istiod-ca-cert\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.513566 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.513548    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-workload-certs\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.514022 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.513985    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istiod-ca-cert\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.514977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.514953    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-envoy\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.515596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.515574    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-podinfo\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.569456 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.569424    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbhcp\" (UniqueName: \"kubernetes.io/projected/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-kube-api-access-mbhcp\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.569603 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.569432    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-token\") pod \"e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") " pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.823200 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.823122    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:46.960220 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:46.960149    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:47.255811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:47.248191    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:47.269344 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:49:47.269316    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca7d913_ef17_4f3a_8e07_bab6ba2048c9.slice/crio-3e3e098f96eeb06b21784e5935241aa548c394b87ab7fa02f8a5c82e4ff93048 WatchSource:0}: Error finding container 3e3e098f96eeb06b21784e5935241aa548c394b87ab7fa02f8a5c82e4ff93048: Status 404 returned error can't find the container with id 3e3e098f96eeb06b21784e5935241aa548c394b87ab7fa02f8a5c82e4ff93048
Oct 31 03:49:47.271497 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:47.271474    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"]
Oct 31 03:49:47.271948 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:47.271921    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:49:47.272001 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:47.271978    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:49:47.272036 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:47.272025    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:49:47.301730 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:47.301703    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:49:47.301849 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:47.301758    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb" podUID="cf89487b-5877-4f0e-81f7-abb1ff44d51a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 03:49:47.398355 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:47.398307    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl" event={"ID":"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9","Type":"ContainerStarted","Data":"b012409c9a55b409ccbfcfbb322d6ec29370694a9e41a37ae541c8ec5f3feb67"}
Oct 31 03:49:47.398543 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:47.398380    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl" event={"ID":"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9","Type":"ContainerStarted","Data":"3e3e098f96eeb06b21784e5935241aa548c394b87ab7fa02f8a5c82e4ff93048"}
Oct 31 03:49:47.517479 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:47.517366    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl" podStartSLOduration=1.5173460749999998 podStartE2EDuration="1.517346075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:47.516243235 +0000 UTC m=+2540.482576594" watchObservedRunningTime="2026-10-31 03:49:47.517346075 +0000 UTC m=+2540.483679435"
Oct 31 03:49:47.824058 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:47.823854    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:47.897866 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:47.897832    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:48.302083 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:48.302047    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:49:48.302298 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:48.302102    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb" podUID="cf89487b-5877-4f0e-81f7-abb1ff44d51a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Oct 31 03:49:48.749245 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:48.749208    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:48.830410 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:48.830383    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:49.305844 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:49.305812    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:49.306097 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:49.306073    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:49.306780 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:49.306761    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:49:49.410990 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:49.410952    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:49.412403 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:49.412381    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:49:50.426560 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:50.426528    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:50.670811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:50.666381    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:53.613838 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:53.613742    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:53.848554 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:53.848521    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:54.260708 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:54.260657    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:56.488682 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:56.488636    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:49:59.318100 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:49:59.318063    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:04.599773 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:04.599741    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:06.988343 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:50:06.988308    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:50:08.241001 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:08.240970    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:08.806848 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:08.806804    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:09.826004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:09.825972    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"]
Oct 31 03:50:09.830068 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:09.830048    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"
Oct 31 03:50:09.882285 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:09.882236    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ca15df\""
Oct 31 03:50:09.933335 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:09.933300    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ca15df-dockercfg-cwvkh\""
Oct 31 03:50:09.937229 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:09.937196    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"]
Oct 31 03:50:09.996859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:09.996821    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mrcj\" (UniqueName: \"kubernetes.io/projected/985cb344-e903-473b-b612-25eeebb99553-kube-api-access-2mrcj\") pod \"payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p\" (UID: \"985cb344-e903-473b-b612-25eeebb99553\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"
Oct 31 03:50:09.997062 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:09.997037    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985cb344-e903-473b-b612-25eeebb99553-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p\" (UID: \"985cb344-e903-473b-b612-25eeebb99553\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"
Oct 31 03:50:10.098405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.098319    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985cb344-e903-473b-b612-25eeebb99553-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p\" (UID: \"985cb344-e903-473b-b612-25eeebb99553\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"
Oct 31 03:50:10.098589 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.098414    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mrcj\" (UniqueName: \"kubernetes.io/projected/985cb344-e903-473b-b612-25eeebb99553-kube-api-access-2mrcj\") pod \"payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p\" (UID: \"985cb344-e903-473b-b612-25eeebb99553\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"
Oct 31 03:50:10.099070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.099048    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985cb344-e903-473b-b612-25eeebb99553-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p\" (UID: \"985cb344-e903-473b-b612-25eeebb99553\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"
Oct 31 03:50:10.156243 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.156214    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mrcj\" (UniqueName: \"kubernetes.io/projected/985cb344-e903-473b-b612-25eeebb99553-kube-api-access-2mrcj\") pod \"payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p\" (UID: \"985cb344-e903-473b-b612-25eeebb99553\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"
Oct 31 03:50:10.443378 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.443304    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"
Oct 31 03:50:10.540436 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.540400    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"]
Oct 31 03:50:10.545510 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.545489    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"
Oct 31 03:50:10.595633 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.595594    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"]
Oct 31 03:50:10.706194 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.706111    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6v9v\" (UniqueName: \"kubernetes.io/projected/70e72a12-774c-40d5-99f9-5b52a6f49710-kube-api-access-c6v9v\") pod \"payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z\" (UID: \"70e72a12-774c-40d5-99f9-5b52a6f49710\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"
Oct 31 03:50:10.706194 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.706158    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e72a12-774c-40d5-99f9-5b52a6f49710-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z\" (UID: \"70e72a12-774c-40d5-99f9-5b52a6f49710\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"
Oct 31 03:50:10.807858 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.807834    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6v9v\" (UniqueName: \"kubernetes.io/projected/70e72a12-774c-40d5-99f9-5b52a6f49710-kube-api-access-c6v9v\") pod \"payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z\" (UID: \"70e72a12-774c-40d5-99f9-5b52a6f49710\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"
Oct 31 03:50:10.808035 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.807881    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e72a12-774c-40d5-99f9-5b52a6f49710-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z\" (UID: \"70e72a12-774c-40d5-99f9-5b52a6f49710\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"
Oct 31 03:50:10.808636 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.808608    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e72a12-774c-40d5-99f9-5b52a6f49710-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z\" (UID: \"70e72a12-774c-40d5-99f9-5b52a6f49710\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"
Oct 31 03:50:10.809160 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:50:10.809133    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985cb344_e903_473b_b612_25eeebb99553.slice/crio-33a3b75485b6efa1389cef7b7848f7e0d2bd00bd9bb725afe7e0cf50d045eeb3 WatchSource:0}: Error finding container 33a3b75485b6efa1389cef7b7848f7e0d2bd00bd9bb725afe7e0cf50d045eeb3: Status 404 returned error can't find the container with id 33a3b75485b6efa1389cef7b7848f7e0d2bd00bd9bb725afe7e0cf50d045eeb3
Oct 31 03:50:10.812940 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.811660    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"]
Oct 31 03:50:10.866945 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:10.866917    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6v9v\" (UniqueName: \"kubernetes.io/projected/70e72a12-774c-40d5-99f9-5b52a6f49710-kube-api-access-c6v9v\") pod \"payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z\" (UID: \"70e72a12-774c-40d5-99f9-5b52a6f49710\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"
Oct 31 03:50:11.167508 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:11.167439    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"
Oct 31 03:50:11.466361 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:11.466268    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:11.530850 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:11.530820    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p" event={"ID":"985cb344-e903-473b-b612-25eeebb99553","Type":"ContainerStarted","Data":"aba88c9bbab6b4ebaff0d80246350e045e7471631e2ab935eabbbcf4acf57530"}
Oct 31 03:50:11.531001 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:11.530880    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"
Oct 31 03:50:11.531001 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:11.530902    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p" event={"ID":"985cb344-e903-473b-b612-25eeebb99553","Type":"ContainerStarted","Data":"33a3b75485b6efa1389cef7b7848f7e0d2bd00bd9bb725afe7e0cf50d045eeb3"}
Oct 31 03:50:11.531762 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:50:11.531737    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e72a12_774c_40d5_99f9_5b52a6f49710.slice/crio-08f2d732dbed9ad0fe8ad81a15bcc2a2317686eb8796aeef9fdf0b4d47cc64fb WatchSource:0}: Error finding container 08f2d732dbed9ad0fe8ad81a15bcc2a2317686eb8796aeef9fdf0b4d47cc64fb: Status 404 returned error can't find the container with id 08f2d732dbed9ad0fe8ad81a15bcc2a2317686eb8796aeef9fdf0b4d47cc64fb
Oct 31 03:50:11.534274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:11.534251    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"]
Oct 31 03:50:11.646983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:11.646927    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p" podStartSLOduration=2.64691475 podStartE2EDuration="2.64691475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:11.645264691 +0000 UTC m=+2564.611598033" watchObservedRunningTime="2026-10-31 03:50:11.64691475 +0000 UTC m=+2564.613248094"
Oct 31 03:50:11.677291 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:11.677263    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:12.538454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:12.538419    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z" event={"ID":"70e72a12-774c-40d5-99f9-5b52a6f49710","Type":"ContainerStarted","Data":"89e75987e1457c3546ad5611d661f21d4ef83a90d7ec19d92a92940d62f549fa"}
Oct 31 03:50:12.538948 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:12.538474    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z" event={"ID":"70e72a12-774c-40d5-99f9-5b52a6f49710","Type":"ContainerStarted","Data":"08f2d732dbed9ad0fe8ad81a15bcc2a2317686eb8796aeef9fdf0b4d47cc64fb"}
Oct 31 03:50:12.538948 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:12.538708    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"
Oct 31 03:50:12.651462 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:12.651412    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z" podStartSLOduration=2.651399842 podStartE2EDuration="2.651399842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:12.651266815 +0000 UTC m=+2565.617600162" watchObservedRunningTime="2026-10-31 03:50:12.651399842 +0000 UTC m=+2565.617733165"
Oct 31 03:50:16.718488 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:16.718448    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:21.627203 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:21.627166    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:22.540937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:22.540910    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"
Oct 31 03:50:23.546956 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:23.546924    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"
Oct 31 03:50:26.157725 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:26.157693    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:26.480245 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:26.480173    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:26.969163 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:26.969112    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:27.171589 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:27.171549    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:27.384678 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:27.384650    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:29.077683 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.077648    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:29.517481 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.517449    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"]
Oct 31 03:50:29.521321 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.521303    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.591834 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.586342    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"]
Oct 31 03:50:29.627006 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.626977    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-93f411-openshift-default-dockercfg-gvbbp\""
Oct 31 03:50:29.647162 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.647108    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-credential-socket\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.647504 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.647464    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-data\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.647636 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.647561    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-token\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.647708 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.647660    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-workload-socket\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.647770 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.647744    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkw7g\" (UniqueName: \"kubernetes.io/projected/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-kube-api-access-bkw7g\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.647886 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.647821    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-envoy\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.647886 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.647852    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-workload-certs\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.648003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.647888    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-podinfo\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.648003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.647920    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istiod-ca-cert\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.693070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.693036    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:29.695560 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.695528    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"]
Oct 31 03:50:29.700048 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.700027    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.749120 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749089    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.749322 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749134    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-workload-socket\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.749322 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749173    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.749322 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749202    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.749322 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749228    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.749322 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749277    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vrlq\" (UniqueName: \"kubernetes.io/projected/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-kube-api-access-8vrlq\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.753610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749325    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkw7g\" (UniqueName: \"kubernetes.io/projected/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-kube-api-access-bkw7g\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.753610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749385    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-envoy\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.753610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749415    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-workload-certs\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.753610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749439    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.753610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749478    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-podinfo\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.753610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749506    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.753610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749532    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istiod-ca-cert\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.753610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749613    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-credential-socket\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.753610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749653    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-data\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.753610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749679    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.753610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749717    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.753610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.749773    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-token\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.753610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.753248    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-workload-socket\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.755377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.755341    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-data\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.755629 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.755607    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-credential-socket\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.755956 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.755931    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"]
Oct 31 03:50:29.756377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.756356    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-workload-certs\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.757289 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.757263    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istiod-ca-cert\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.758497 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.758472    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-envoy\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.758645 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.758621    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-podinfo\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.807709 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.807636    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-token\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.810662 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.810638    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkw7g\" (UniqueName: \"kubernetes.io/projected/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-kube-api-access-bkw7g\") pod \"e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") " pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.827950 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.827929    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b59563-openshift-default-dockercfg-h4q7c\""
Oct 31 03:50:29.834156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.834137    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:29.850834 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.850809    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.850939 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.850886    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.850939 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.850910    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.851038 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.850992    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.851038 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.851027    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.851148 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.851054    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.851254 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.851230    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.851298 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.851286    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-credential-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.851335 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.851304    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrlq\" (UniqueName: \"kubernetes.io/projected/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-kube-api-access-8vrlq\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.851373 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.851345    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-data\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.851449 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.851424    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.851874 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.851850    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-workload-socket\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.852076 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.852048    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istiod-ca-cert\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.852200 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.852084    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-workload-certs\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.853476 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.853430    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-envoy\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.853862 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.853842    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-podinfo\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.912906 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.912039    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-token\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:29.912906 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:29.912596    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrlq\" (UniqueName: \"kubernetes.io/projected/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-kube-api-access-8vrlq\") pod \"e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") " pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:30.014868 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.014833    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:30.096283 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.096252    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:30.257628 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:50:30.257583    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e8d2a4_3ce9_44f3_892f_d23a0dd55ac2.slice/crio-64e7e31dbab15930d67f38537a106bbc402e83f0c5f591a64ce8e8d8a2ef5e97 WatchSource:0}: Error finding container 64e7e31dbab15930d67f38537a106bbc402e83f0c5f591a64ce8e8d8a2ef5e97: Status 404 returned error can't find the container with id 64e7e31dbab15930d67f38537a106bbc402e83f0c5f591a64ce8e8d8a2ef5e97
Oct 31 03:50:30.260105 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.260070    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:30.260221 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.260150    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:30.260221 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.260188    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"]
Oct 31 03:50:30.260221 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.260195    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:30.388671 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:50:30.388643    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d719ccc_8bf1_4e14_a62e_0dcea593ed7a.slice/crio-81d493542dd79fb844556ddb801e94547acfc4c69c56d540f5ed4733a60ece68 WatchSource:0}: Error finding container 81d493542dd79fb844556ddb801e94547acfc4c69c56d540f5ed4733a60ece68: Status 404 returned error can't find the container with id 81d493542dd79fb844556ddb801e94547acfc4c69c56d540f5ed4733a60ece68
Oct 31 03:50:30.392391 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.392323    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:30.392600 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.392573    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:30.392732 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.392703    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:30.394821 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.394765    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"]
Oct 31 03:50:30.654585 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.654481    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm" event={"ID":"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a","Type":"ContainerStarted","Data":"4721740c281a96ca29d3adb400b4060fc513d1aac334e3fe077b221b4c82eedf"}
Oct 31 03:50:30.654585 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.654545    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm" event={"ID":"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a","Type":"ContainerStarted","Data":"81d493542dd79fb844556ddb801e94547acfc4c69c56d540f5ed4733a60ece68"}
Oct 31 03:50:30.656234 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.656210    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt" event={"ID":"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2","Type":"ContainerStarted","Data":"6af4eb84d7cb6c01d01e23a798aec50ffdf0df17716261749a2349c673e8f0d2"}
Oct 31 03:50:30.656369 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.656248    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt" event={"ID":"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2","Type":"ContainerStarted","Data":"64e7e31dbab15930d67f38537a106bbc402e83f0c5f591a64ce8e8d8a2ef5e97"}
Oct 31 03:50:30.774655 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.774596    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm" podStartSLOduration=1.774577872 podStartE2EDuration="1.774577872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:30.772713995 +0000 UTC m=+2583.739047340" watchObservedRunningTime="2026-10-31 03:50:30.774577872 +0000 UTC m=+2583.740911219"
Oct 31 03:50:30.834983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.834946    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:30.890353 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:30.890301    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt" podStartSLOduration=1.8902878300000001 podStartE2EDuration="1.89028783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:30.888674161 +0000 UTC m=+2583.855007504" watchObservedRunningTime="2026-10-31 03:50:30.89028783 +0000 UTC m=+2583.856621174"
Oct 31 03:50:31.015347 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:31.015307    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:31.017241 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:31.017211    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:50:31.017366 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:31.017275    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm" podUID="7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 31 03:50:31.752926 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:31.747319    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:31.863271 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:31.863239    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:32.016198 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:32.016113    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:50:32.016198 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:32.016167    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm" podUID="7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 31 03:50:32.059523 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:32.059482    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:32.355868 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:32.352389    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:32.679865 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:32.673742    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:32.679865 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:32.675124    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:50:33.015439 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:33.015392    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:50:33.016018 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:33.015458    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm" podUID="7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 31 03:50:33.767042 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:33.767004    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:34.019726 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:34.019202    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:50:34.019726 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:34.019272    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm" podUID="7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 31 03:50:34.676957 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:34.676904    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:50:34.684905 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:34.684864    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:50:34.715265 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:34.715217    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded"
Oct 31 03:50:34.786950 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:34.786900    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:50:35.020811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:35.020315    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:50:35.020811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:35.020411    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm" podUID="7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 31 03:50:35.370387 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:35.370298    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:35.595663 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:35.595632    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:35.863074 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:35.863040    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:36.015276 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.015236    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:50:36.015470 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.015307    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm" podUID="7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 31 03:50:36.109446 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.109413    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:36.226469 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.226390    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"]
Oct 31 03:50:36.230645 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.230628    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"
Oct 31 03:50:36.277829 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.277754    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt9pj\" (UniqueName: \"kubernetes.io/projected/8c101703-159d-4ae6-8e41-e669698c9130-kube-api-access-gt9pj\") pod \"payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6\" (UID: \"8c101703-159d-4ae6-8e41-e669698c9130\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"
Oct 31 03:50:36.278037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.277952    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c101703-159d-4ae6-8e41-e669698c9130-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6\" (UID: \"8c101703-159d-4ae6-8e41-e669698c9130\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"
Oct 31 03:50:36.294106 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.294074    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2339ed\""
Oct 31 03:50:36.379832 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.370354    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2339ed-dockercfg-nq89p\""
Oct 31 03:50:36.379832 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.376972    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"]
Oct 31 03:50:36.379832 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.379406    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt9pj\" (UniqueName: \"kubernetes.io/projected/8c101703-159d-4ae6-8e41-e669698c9130-kube-api-access-gt9pj\") pod \"payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6\" (UID: \"8c101703-159d-4ae6-8e41-e669698c9130\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"
Oct 31 03:50:36.379832 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.379510    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c101703-159d-4ae6-8e41-e669698c9130-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6\" (UID: \"8c101703-159d-4ae6-8e41-e669698c9130\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"
Oct 31 03:50:36.380497 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.380469    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c101703-159d-4ae6-8e41-e669698c9130-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6\" (UID: \"8c101703-159d-4ae6-8e41-e669698c9130\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"
Oct 31 03:50:36.468265 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.468230    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt9pj\" (UniqueName: \"kubernetes.io/projected/8c101703-159d-4ae6-8e41-e669698c9130-kube-api-access-gt9pj\") pod \"payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6\" (UID: \"8c101703-159d-4ae6-8e41-e669698c9130\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"
Oct 31 03:50:36.570173 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.570137    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"
Oct 31 03:50:36.960319 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:36.960082    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"]
Oct 31 03:50:36.992774 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:50:36.992740    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:50:37.015656 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:37.015625    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:50:37.015765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:37.015685    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm" podUID="7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 31 03:50:37.741304 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:37.741244    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6" event={"ID":"8c101703-159d-4ae6-8e41-e669698c9130","Type":"ContainerStarted","Data":"12260331744c799f43c0b87b8144d6e63f1990a5780ef863dd204d917e3794bc"}
Oct 31 03:50:37.741897 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:37.741875    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"
Oct 31 03:50:37.741981 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:37.741910    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6" event={"ID":"8c101703-159d-4ae6-8e41-e669698c9130","Type":"ContainerStarted","Data":"af3e603eae7e165d913de69c894085ce62bd2198c031ecbedce109b991c209bf"}
Oct 31 03:50:37.856402 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:37.856323    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6" podStartSLOduration=1.856304204 podStartE2EDuration="1.856304204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:37.854003085 +0000 UTC m=+2590.820336430" watchObservedRunningTime="2026-10-31 03:50:37.856304204 +0000 UTC m=+2590.822637550"
Oct 31 03:50:38.015602 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:38.015513    4900 patch_prober.go:29] interesting pod/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:50:38.015771 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:38.015589    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm" podUID="7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 31 03:50:39.629225 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:39.629187    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:39.907655 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:39.907577    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:39.907925 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:39.907905    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:39.908733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:39.908712    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:50:40.514562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:40.514529    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:40.723008 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:40.722976    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:40.849589 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:40.849511    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"]
Oct 31 03:50:40.854423 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:40.854396    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"
Oct 31 03:50:40.904580 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:40.904553    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"]
Oct 31 03:50:40.907821 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:40.907782    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-93f411\""
Oct 31 03:50:40.907975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:40.907956    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-93f411-dockercfg-hp2m4\""
Oct 31 03:50:40.940129 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:40.940078    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbcpf\" (UniqueName: \"kubernetes.io/projected/c0b7639f-ab0a-4634-9c22-c6408daeb5d0-kube-api-access-cbcpf\") pod \"payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6\" (UID: \"c0b7639f-ab0a-4634-9c22-c6408daeb5d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"
Oct 31 03:50:40.943979 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:40.943943    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b7639f-ab0a-4634-9c22-c6408daeb5d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6\" (UID: \"c0b7639f-ab0a-4634-9c22-c6408daeb5d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"
Oct 31 03:50:41.003143 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.003110    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"]
Oct 31 03:50:41.007136 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.007114    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"
Oct 31 03:50:41.045468 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.045429    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbcpf\" (UniqueName: \"kubernetes.io/projected/c0b7639f-ab0a-4634-9c22-c6408daeb5d0-kube-api-access-cbcpf\") pod \"payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6\" (UID: \"c0b7639f-ab0a-4634-9c22-c6408daeb5d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"
Oct 31 03:50:41.045608 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.045491    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b7639f-ab0a-4634-9c22-c6408daeb5d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6\" (UID: \"c0b7639f-ab0a-4634-9c22-c6408daeb5d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"
Oct 31 03:50:41.045608 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.045542    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rgt\" (UniqueName: \"kubernetes.io/projected/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7-kube-api-access-l4rgt\") pod \"payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng\" (UID: \"0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"
Oct 31 03:50:41.045608 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.045583    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng\" (UID: \"0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"
Oct 31 03:50:41.046371 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.046348    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b7639f-ab0a-4634-9c22-c6408daeb5d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6\" (UID: \"c0b7639f-ab0a-4634-9c22-c6408daeb5d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"
Oct 31 03:50:41.064172 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.064143    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"]
Oct 31 03:50:41.102513 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.102441    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbcpf\" (UniqueName: \"kubernetes.io/projected/c0b7639f-ab0a-4634-9c22-c6408daeb5d0-kube-api-access-cbcpf\") pod \"payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6\" (UID: \"c0b7639f-ab0a-4634-9c22-c6408daeb5d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"
Oct 31 03:50:41.146982 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.146946    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rgt\" (UniqueName: \"kubernetes.io/projected/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7-kube-api-access-l4rgt\") pod \"payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng\" (UID: \"0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"
Oct 31 03:50:41.147161 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.147014    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng\" (UID: \"0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"
Oct 31 03:50:41.147688 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.147662    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng\" (UID: \"0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"
Oct 31 03:50:41.171483 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.171457    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"
Oct 31 03:50:41.204059 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.204027    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rgt\" (UniqueName: \"kubernetes.io/projected/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7-kube-api-access-l4rgt\") pod \"payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng\" (UID: \"0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"
Oct 31 03:50:41.323570 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.323537    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"
Oct 31 03:50:41.541842 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:50:41.541808    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b7639f_ab0a_4634_9c22_c6408daeb5d0.slice/crio-757a23c20db6ee72ac9b94329da2ab90306c0a1d57ec65d3f9241e8f41819ed6 WatchSource:0}: Error finding container 757a23c20db6ee72ac9b94329da2ab90306c0a1d57ec65d3f9241e8f41819ed6: Status 404 returned error can't find the container with id 757a23c20db6ee72ac9b94329da2ab90306c0a1d57ec65d3f9241e8f41819ed6
Oct 31 03:50:41.545112 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.545088    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"]
Oct 31 03:50:41.692553 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:50:41.692526    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c361df3_1ec3_42cc_bd4a_b1f9c659bcb7.slice/crio-1d8b1eb2e417a5324538872a4f068f76d9d0fb037e2b6d26b0d2b5979fda35fa WatchSource:0}: Error finding container 1d8b1eb2e417a5324538872a4f068f76d9d0fb037e2b6d26b0d2b5979fda35fa: Status 404 returned error can't find the container with id 1d8b1eb2e417a5324538872a4f068f76d9d0fb037e2b6d26b0d2b5979fda35fa
Oct 31 03:50:41.695446 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.695423    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"]
Oct 31 03:50:41.769471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.769440    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6" event={"ID":"c0b7639f-ab0a-4634-9c22-c6408daeb5d0","Type":"ContainerStarted","Data":"5ec7b7d965b540dfa1dc602036ed07b8fda38965cff22f44c90b709e9205506d"}
Oct 31 03:50:41.769830 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.769504    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6" event={"ID":"c0b7639f-ab0a-4634-9c22-c6408daeb5d0","Type":"ContainerStarted","Data":"757a23c20db6ee72ac9b94329da2ab90306c0a1d57ec65d3f9241e8f41819ed6"}
Oct 31 03:50:41.769830 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.769633    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"
Oct 31 03:50:41.771284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:41.771254    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng" event={"ID":"0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7","Type":"ContainerStarted","Data":"1d8b1eb2e417a5324538872a4f068f76d9d0fb037e2b6d26b0d2b5979fda35fa"}
Oct 31 03:50:42.777519 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:42.777483    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng" event={"ID":"0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7","Type":"ContainerStarted","Data":"9635719a59d69ae4f3581c6428f26a953b8b636660a8cb2e96938ae3fa02c5fc"}
Oct 31 03:50:42.777958 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:42.777638    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"
Oct 31 03:50:42.891570 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:42.891520    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng" podStartSLOduration=2.891505735 podStartE2EDuration="2.891505735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:42.891071535 +0000 UTC m=+2595.857404879" watchObservedRunningTime="2026-10-31 03:50:42.891505735 +0000 UTC m=+2595.857839113"
Oct 31 03:50:42.892537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:42.892499    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6" podStartSLOduration=2.892487404 podStartE2EDuration="2.892487404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:41.882483936 +0000 UTC m=+2594.848817279" watchObservedRunningTime="2026-10-31 03:50:42.892487404 +0000 UTC m=+2595.858820772"
Oct 31 03:50:44.752493 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.752460    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:44.790528 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.790494    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"]
Oct 31 03:50:44.794694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.794675    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:44.840866 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.840832    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"]
Oct 31 03:50:44.844658 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.844637    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:44.876518 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.876492    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"]
Oct 31 03:50:44.902655 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.902622    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:44.902874 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.902687    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:44.902874 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.902737    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:44.903009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.902891    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:44.903009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.902943    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm8rl\" (UniqueName: \"kubernetes.io/projected/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-kube-api-access-pm8rl\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:44.903009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.902967    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:44.903009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.902993    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:44.903226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.903013    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:44.903226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.903081    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:44.903226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.903118    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:44.903226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.903148    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgjmq\" (UniqueName: \"kubernetes.io/projected/b38b8385-a404-433d-bcee-5e5ffb33d5da-kube-api-access-tgjmq\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:44.903226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.903182    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:44.903226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.903218    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:44.903549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.903289    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:44.903549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.903317    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:44.903549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.903343    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:44.903549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.903406    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:44.903549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.903451    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38b8385-a404-433d-bcee-5e5ffb33d5da-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:44.905256 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.905232    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"]
Oct 31 03:50:44.967918 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:44.967888    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:45.005266 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005189    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.005266 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005246    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.005506 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005274    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.005506 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005314    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.005506 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005343    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm8rl\" (UniqueName: \"kubernetes.io/projected/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-kube-api-access-pm8rl\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.005506 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005479    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.005756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005559    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.005756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005612    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.005756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005632    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.005756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005683    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.005756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005692    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.005756 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005689    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.006130 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005770    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.006130 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005845    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgjmq\" (UniqueName: \"kubernetes.io/projected/b38b8385-a404-433d-bcee-5e5ffb33d5da-kube-api-access-tgjmq\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.006130 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005881    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.006130 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005892    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.006130 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.005958    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.006130 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.006009    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.006130 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.006054    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.006130 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.006098    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.006571 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.006136    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.006571 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.006170    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38b8385-a404-433d-bcee-5e5ffb33d5da-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.006571 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.006175    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.006571 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.006212    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.006571 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.006356    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.006800 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.006608    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.006800 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.006636    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.007015 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.006984    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38b8385-a404-433d-bcee-5e5ffb33d5da-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.008166 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.008143    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.008326 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.008304    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.008372 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.008319    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.008935 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.008919    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.062851 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.062813    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.062990 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.062926    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.063084 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.063063    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm8rl\" (UniqueName: \"kubernetes.io/projected/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-kube-api-access-pm8rl\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.063178 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.063164    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgjmq\" (UniqueName: \"kubernetes.io/projected/b38b8385-a404-433d-bcee-5e5ffb33d5da-kube-api-access-tgjmq\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.108828 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.108807    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:45.162071 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.162032    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:45.536623 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.536588    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"]
Oct 31 03:50:45.570623 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:50:45.570597    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38b8385_a404_433d_bcee_5e5ffb33d5da.slice/crio-e377a5d2e1f7f5beaf9af5ff4ff2ce49820cdb31ff546412417fca4caa4c11f8 WatchSource:0}: Error finding container e377a5d2e1f7f5beaf9af5ff4ff2ce49820cdb31ff546412417fca4caa4c11f8: Status 404 returned error can't find the container with id e377a5d2e1f7f5beaf9af5ff4ff2ce49820cdb31ff546412417fca4caa4c11f8
Oct 31 03:50:45.573196 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.573163    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:45.573256 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.573234    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:45.573296 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.573269    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:45.583041 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:50:45.583009    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a6cc6d_73ae_42a5_899e_c8c05bf010f3.slice/crio-8b3447c8586402aa07b571435fe6e8bc37cc734bbafe07bd2dd53f2e7c0a5aa4 WatchSource:0}: Error finding container 8b3447c8586402aa07b571435fe6e8bc37cc734bbafe07bd2dd53f2e7c0a5aa4: Status 404 returned error can't find the container with id 8b3447c8586402aa07b571435fe6e8bc37cc734bbafe07bd2dd53f2e7c0a5aa4
Oct 31 03:50:45.585494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.585455    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:45.585601 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.585535    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:45.585601 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.585580    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:45.586324 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.586302    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"]
Oct 31 03:50:45.799688 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.799593    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd" event={"ID":"44a6cc6d-73ae-42a5-899e-c8c05bf010f3","Type":"ContainerStarted","Data":"29918f8d23744016edd5b764b1b41f205240fd386218fa2686a0db6c182feda1"}
Oct 31 03:50:45.799688 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.799654    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd" event={"ID":"44a6cc6d-73ae-42a5-899e-c8c05bf010f3","Type":"ContainerStarted","Data":"8b3447c8586402aa07b571435fe6e8bc37cc734bbafe07bd2dd53f2e7c0a5aa4"}
Oct 31 03:50:45.801627 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.801597    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn" event={"ID":"b38b8385-a404-433d-bcee-5e5ffb33d5da","Type":"ContainerStarted","Data":"86761543486b7d4644fe8db9d0004575d2574f60a12446b38f9e990eca30e2e8"}
Oct 31 03:50:45.801781 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.801650    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn" event={"ID":"b38b8385-a404-433d-bcee-5e5ffb33d5da","Type":"ContainerStarted","Data":"e377a5d2e1f7f5beaf9af5ff4ff2ce49820cdb31ff546412417fca4caa4c11f8"}
Oct 31 03:50:45.920843 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:45.920764    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd" podStartSLOduration=1.9207456760000001 podStartE2EDuration="1.920745676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:45.916698543 +0000 UTC m=+2598.883031869" watchObservedRunningTime="2026-10-31 03:50:45.920745676 +0000 UTC m=+2598.887079019"
Oct 31 03:50:46.037798 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:46.037725    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn" podStartSLOduration=2.037710384 podStartE2EDuration="2.037710384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:46.03474373 +0000 UTC m=+2599.001077075" watchObservedRunningTime="2026-10-31 03:50:46.037710384 +0000 UTC m=+2599.004043728"
Oct 31 03:50:46.113745 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:46.113648    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:46.115487 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:46.115447    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 03:50:46.115661 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:46.115509    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn" podUID="b38b8385-a404-433d-bcee-5e5ffb33d5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 03:50:46.167812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:46.164706    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:46.167812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:46.165038    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 03:50:46.167812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:46.165090    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd" podUID="44a6cc6d-73ae-42a5-899e-c8c05bf010f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 03:50:47.110033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:47.109995    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 03:50:47.110433 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:47.110055    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn" podUID="b38b8385-a404-433d-bcee-5e5ffb33d5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 03:50:47.168215 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:47.168180    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 03:50:47.168495 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:47.168467    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd" podUID="44a6cc6d-73ae-42a5-899e-c8c05bf010f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 03:50:48.109400 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:48.109365    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 03:50:48.109556 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:48.109421    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn" podUID="b38b8385-a404-433d-bcee-5e5ffb33d5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 03:50:48.167812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:48.166782    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 03:50:48.167812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:48.166860    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd" podUID="44a6cc6d-73ae-42a5-899e-c8c05bf010f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 03:50:48.785857 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:48.784007    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"
Oct 31 03:50:49.109167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:49.109074    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 03:50:49.109167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:49.109132    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn" podUID="b38b8385-a404-433d-bcee-5e5ffb33d5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 03:50:49.185043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:49.183912    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 03:50:49.185043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:49.183971    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd" podUID="44a6cc6d-73ae-42a5-899e-c8c05bf010f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 03:50:50.107575 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:50.107543    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:50.109694 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:50.109671    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 03:50:50.109806 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:50.109726    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn" podUID="b38b8385-a404-433d-bcee-5e5ffb33d5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 03:50:50.167624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:50.167516    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 03:50:50.167624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:50.167570    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd" podUID="44a6cc6d-73ae-42a5-899e-c8c05bf010f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 03:50:50.956924 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:50.956872    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:51.110006 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.109970    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 03:50:51.110172 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.110020    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn" podUID="b38b8385-a404-433d-bcee-5e5ffb33d5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 03:50:51.169813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.169709    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 03:50:51.169813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.169765    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd" podUID="44a6cc6d-73ae-42a5-899e-c8c05bf010f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 03:50:51.490999 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.488738    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"]
Oct 31 03:50:51.497253 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.497223    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.513610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.513437    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"]
Oct 31 03:50:51.614877 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.614846    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-861dfd-openshift-default-dockercfg-p7zw2\""
Oct 31 03:50:51.619943 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.619912    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:51.620588 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.620560    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-envoy\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.620697 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.620615    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-podinfo\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.620770 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.620695    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba5a14c-8608-4005-961d-869f98e5d3a7-istiod-ca-cert\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.620770 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.620755    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-data\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.620898 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.620822    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcq7h\" (UniqueName: \"kubernetes.io/projected/fba5a14c-8608-4005-961d-869f98e5d3a7-kube-api-access-wcq7h\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.620898 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.620862    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-credential-socket\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.620991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.620921    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-workload-socket\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.621104 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.621029    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-workload-certs\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.621104 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.621056    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-token\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.722551 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.722514    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-envoy\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.722755 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.722576    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-podinfo\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.722755 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.722650    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba5a14c-8608-4005-961d-869f98e5d3a7-istiod-ca-cert\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.722755 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.722715    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-data\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.722755 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.722752    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcq7h\" (UniqueName: \"kubernetes.io/projected/fba5a14c-8608-4005-961d-869f98e5d3a7-kube-api-access-wcq7h\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.723002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.722805    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-credential-socket\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.723002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.722863    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-workload-socket\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.723143 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.723115    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-data\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.723268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.723244    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-workload-certs\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.723334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.723280    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-token\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.723426 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.723396    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-credential-socket\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.723574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.723482    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba5a14c-8608-4005-961d-869f98e5d3a7-istiod-ca-cert\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.723574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.723526    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-workload-socket\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.723574 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.723544    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-workload-certs\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.725422 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.725400    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-envoy\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.725909 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.725886    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-podinfo\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.780728 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.780647    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-token\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.782299 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.782275    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcq7h\" (UniqueName: \"kubernetes.io/projected/fba5a14c-8608-4005-961d-869f98e5d3a7-kube-api-access-wcq7h\") pod \"e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") " pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.830800 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.830764    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:51.881213 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:51.881176    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:52.109448 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:52.109371    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 03:50:52.109448 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:52.109431    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn" podUID="b38b8385-a404-433d-bcee-5e5ffb33d5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 03:50:52.163500 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:52.163449    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Oct 31 03:50:52.163695 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:52.163510    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd" podUID="44a6cc6d-73ae-42a5-899e-c8c05bf010f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Oct 31 03:50:52.256763 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:50:52.256726    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba5a14c_8608_4005_961d_869f98e5d3a7.slice/crio-14b0d754c998f85c44289e46ba053c6994b2f2749e67c77bf1d4377a4d42d1be WatchSource:0}: Error finding container 14b0d754c998f85c44289e46ba053c6994b2f2749e67c77bf1d4377a4d42d1be: Status 404 returned error can't find the container with id 14b0d754c998f85c44289e46ba053c6994b2f2749e67c77bf1d4377a4d42d1be
Oct 31 03:50:52.259580 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:52.259554    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"]
Oct 31 03:50:52.259884 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:52.259848    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:52.259954 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:52.259934    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:52.260002 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:52.259985    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:52.782684 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:52.782645    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"
Oct 31 03:50:52.848040 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:52.847993    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2" event={"ID":"fba5a14c-8608-4005-961d-869f98e5d3a7","Type":"ContainerStarted","Data":"a1994aebfe40728316c650405921475a95188999833b35fddc88feb3a6451a16"}
Oct 31 03:50:52.848244 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:52.848063    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2" event={"ID":"fba5a14c-8608-4005-961d-869f98e5d3a7","Type":"ContainerStarted","Data":"14b0d754c998f85c44289e46ba053c6994b2f2749e67c77bf1d4377a4d42d1be"}
Oct 31 03:50:53.017941 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:53.017873    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2" podStartSLOduration=2.017851394 podStartE2EDuration="2.017851394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:53.016269908 +0000 UTC m=+2605.982603255" watchObservedRunningTime="2026-10-31 03:50:53.017851394 +0000 UTC m=+2605.984184751"
Oct 31 03:50:53.165152 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:53.165066    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:53.165611 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:53.165560    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:53.166913 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:53.166893    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:50:53.193116 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:53.193084    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:53.194210 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:53.194183    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:53.195036 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:53.195015    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:50:53.787003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:53.786971    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"
Oct 31 03:50:53.831007 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:53.830980    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:53.836367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:53.836343    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:53.854112 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:53.854083    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:53.855119 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:53.855098    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:50:55.332862 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:55.332770    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:56.010128 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:56.010064    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:56.356606 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:56.356522    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:56.827047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:56.827015    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:59.292385 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:59.292353    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:59.505045 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:59.505014    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:50:59.783107 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:59.783057    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"]
Oct 31 03:50:59.804332 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:59.804301    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"
Oct 31 03:50:59.838370 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:59.837774    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca32c87f-ca00-4404-9560-22463f06c330-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp\" (UID: \"ca32c87f-ca00-4404-9560-22463f06c330\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"
Oct 31 03:50:59.838370 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:59.838046    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k84xx\" (UniqueName: \"kubernetes.io/projected/ca32c87f-ca00-4404-9560-22463f06c330-kube-api-access-k84xx\") pod \"payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp\" (UID: \"ca32c87f-ca00-4404-9560-22463f06c330\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"
Oct 31 03:50:59.842206 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:59.842178    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"]
Oct 31 03:50:59.886381 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:59.884072    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-861dfd\""
Oct 31 03:50:59.886381 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:59.884173    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-861dfd-dockercfg-vkfpf\""
Oct 31 03:50:59.940329 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:59.940294    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k84xx\" (UniqueName: \"kubernetes.io/projected/ca32c87f-ca00-4404-9560-22463f06c330-kube-api-access-k84xx\") pod \"payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp\" (UID: \"ca32c87f-ca00-4404-9560-22463f06c330\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"
Oct 31 03:50:59.940540 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:59.940480    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca32c87f-ca00-4404-9560-22463f06c330-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp\" (UID: \"ca32c87f-ca00-4404-9560-22463f06c330\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"
Oct 31 03:50:59.941380 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:59.941351    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca32c87f-ca00-4404-9560-22463f06c330-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp\" (UID: \"ca32c87f-ca00-4404-9560-22463f06c330\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"
Oct 31 03:50:59.993465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:50:59.993430    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k84xx\" (UniqueName: \"kubernetes.io/projected/ca32c87f-ca00-4404-9560-22463f06c330-kube-api-access-k84xx\") pod \"payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp\" (UID: \"ca32c87f-ca00-4404-9560-22463f06c330\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"
Oct 31 03:51:00.135810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:00.135713    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"
Oct 31 03:51:00.505122 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:00.505088    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"]
Oct 31 03:51:00.505913 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:51:00.505888    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca32c87f_ca00_4404_9560_22463f06c330.slice/crio-fb55c935a5562a3f15675c672b198811eac98df42be92266171365c4aa44204a WatchSource:0}: Error finding container fb55c935a5562a3f15675c672b198811eac98df42be92266171365c4aa44204a: Status 404 returned error can't find the container with id fb55c935a5562a3f15675c672b198811eac98df42be92266171365c4aa44204a
Oct 31 03:51:00.903502 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:00.903409    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp" event={"ID":"ca32c87f-ca00-4404-9560-22463f06c330","Type":"ContainerStarted","Data":"3d75725344d0bbb75171871e3186f01aea2e1fcfc31a51ccb4fb0fe00477a69d"}
Oct 31 03:51:00.903502 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:00.903471    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp" event={"ID":"ca32c87f-ca00-4404-9560-22463f06c330","Type":"ContainerStarted","Data":"fb55c935a5562a3f15675c672b198811eac98df42be92266171365c4aa44204a"}
Oct 31 03:51:00.903502 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:00.903499    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"
Oct 31 03:51:01.011594 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:01.011541    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp" podStartSLOduration=2.011527003 podStartE2EDuration="2.011527003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:51:01.009979022 +0000 UTC m=+2613.976312366" watchObservedRunningTime="2026-10-31 03:51:01.011527003 +0000 UTC m=+2613.977860347"
Oct 31 03:51:01.081336 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:01.081287    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:03.285986 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:03.285938    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:51:03.358024 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:03.357969    4900 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:51:03.382952 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:03.382907    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:51:06.988126 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:51:06.988093    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:51:08.794840 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:08.794804    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:11.913486 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:11.913456    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"
Oct 31 03:51:12.011842 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:12.011813    4900 generic.go:356] "Generic (PLEG): container finished" podID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" containerID="b012409c9a55b409ccbfcfbb322d6ec29370694a9e41a37ae541c8ec5f3feb67" exitCode=0
Oct 31 03:51:12.011996 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:12.011887    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl" event={"ID":"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9","Type":"ContainerDied","Data":"b012409c9a55b409ccbfcfbb322d6ec29370694a9e41a37ae541c8ec5f3feb67"}
Oct 31 03:51:12.012308 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:12.012294    4900 scope.go:122] "RemoveContainer" containerID="b012409c9a55b409ccbfcfbb322d6ec29370694a9e41a37ae541c8ec5f3feb67"
Oct 31 03:51:12.013809 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:12.013766    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:51:12.013865 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:12.013850    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:51:12.013900 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:12.013884    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:51:13.019646 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:13.019611    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl" event={"ID":"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9","Type":"ContainerStarted","Data":"bfe0122fd5106a6e3cda160d76462e5e26a366032bf79123a7358e69afc71649"}
Oct 31 03:51:13.778562 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:13.776762    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:13.823650 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:13.823613    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:51:13.823859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:13.823761    4900 patch_prober.go:29] interesting pod/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:51:13.823859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:13.823837    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl" podUID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 31 03:51:14.823541 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:14.823497    4900 patch_prober.go:29] interesting pod/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:51:14.824003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:14.823559    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl" podUID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 31 03:51:15.823876 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:15.823834    4900 patch_prober.go:29] interesting pod/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:51:15.824335 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:15.823892    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl" podUID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 31 03:51:16.823898 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:16.823859    4900 patch_prober.go:29] interesting pod/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:51:16.824298 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:16.823915    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl" podUID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 31 03:51:17.824152 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:17.824113    4900 patch_prober.go:29] interesting pod/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 31 03:51:17.824565 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:17.824185    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl" podUID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 31 03:51:18.637025 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:18.636990    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:18.828884 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:18.828852    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:51:18.829288 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:18.829150    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:51:18.829828 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:18.829811    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:51:23.468329 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:23.468291    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:28.219295 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:28.219261    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:28.432610 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:28.432519    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:28.646116 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:28.646077    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:32.700981 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:32.700947    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:33.268101 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:33.268016    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:33.474670 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:33.474633    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:34.477997 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:34.477936    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:34.698888 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:34.698849    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:36.984956 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:51:36.984921    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:51:39.530211 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:39.530181    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:40.118192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:40.118155    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:40.316270 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:40.316235    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:41.771446 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:41.771395    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:42.419811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:42.412754    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:44.877073 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:44.877042    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:48.906919 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:48.906881    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:50.502971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:50.502939    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:53.285217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:53.285188    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:54.971735 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:54.971701    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:56.479728 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:56.479645    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:57.176754 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:57.176719    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:57.319154 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:57.319107    4900 generic.go:356] "Generic (PLEG): container finished" podID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerID="6af4eb84d7cb6c01d01e23a798aec50ffdf0df17716261749a2349c673e8f0d2" exitCode=0
Oct 31 03:51:57.319342 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:57.319182    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt" event={"ID":"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2","Type":"ContainerDied","Data":"6af4eb84d7cb6c01d01e23a798aec50ffdf0df17716261749a2349c673e8f0d2"}
Oct 31 03:51:57.319638 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:57.319619    4900 scope.go:122] "RemoveContainer" containerID="6af4eb84d7cb6c01d01e23a798aec50ffdf0df17716261749a2349c673e8f0d2"
Oct 31 03:51:57.321302 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:57.321276    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:51:57.321368 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:57.321344    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:51:57.321410 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:57.321377    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:51:57.394304 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:57.394276    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:51:58.326842 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:58.326777    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt" event={"ID":"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2","Type":"ContainerStarted","Data":"7949a4a247db97bf76ddc6aea91b560f0697e951cc38f71a272a7c619d9c6f1f"}
Oct 31 03:51:58.841460 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:58.841422    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:51:58.841888 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:58.841864    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:51:58.842086 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:58.842061    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt" podUID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 03:51:59.854812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:59.852229    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:51:59.854812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:51:59.852310    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt" podUID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 03:52:00.854813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:00.848253    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:52:00.854813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:00.848321    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt" podUID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 03:52:01.022428 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:01.022398    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:01.500485 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:01.500452    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:01.852870 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:01.851224    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:52:01.852870 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:01.851293    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt" podUID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 03:52:02.673781 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:02.673729    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:52:02.853814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:02.840204    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:52:02.853814 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:02.840287    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt" podUID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 03:52:03.838352 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:03.838321    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:52:03.839505 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:03.839485    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 03:52:05.429580 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:05.429543    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:05.644353 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:05.644303    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:06.085531 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:06.085500    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:06.511517 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:06.511480    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"]
Oct 31 03:52:06.512053 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:06.511729    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z" podUID="70e72a12-774c-40d5-99f9-5b52a6f49710" containerName="payload-processing" containerID="cri-o://89e75987e1457c3546ad5611d661f21d4ef83a90d7ec19d92a92940d62f549fa" gracePeriod=30
Oct 31 03:52:06.632819 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:06.618914    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"]
Oct 31 03:52:06.632819 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:06.631902    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p" podUID="985cb344-e903-473b-b612-25eeebb99553" containerName="payload-pre-processing" containerID="cri-o://aba88c9bbab6b4ebaff0d80246350e045e7471631e2ab935eabbbcf4acf57530" gracePeriod=30
Oct 31 03:52:06.753096 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:06.752510    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:06.818923 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:06.818772    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"
Oct 31 03:52:06.896409 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:06.896387    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"
Oct 31 03:52:06.913445 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:06.913420    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70e72a12-774c-40d5-99f9-5b52a6f49710-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e72a12-774c-40d5-99f9-5b52a6f49710-plugins-config-volume\") pod \"70e72a12-774c-40d5-99f9-5b52a6f49710\" (UID: \"70e72a12-774c-40d5-99f9-5b52a6f49710\") "
Oct 31 03:52:06.913662 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:06.913645    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70e72a12-774c-40d5-99f9-5b52a6f49710-kube-api-access-c6v9v\" (UniqueName: \"kubernetes.io/projected/70e72a12-774c-40d5-99f9-5b52a6f49710-kube-api-access-c6v9v\") pod \"70e72a12-774c-40d5-99f9-5b52a6f49710\" (UID: \"70e72a12-774c-40d5-99f9-5b52a6f49710\") "
Oct 31 03:52:06.913853 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:06.913830    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e72a12-774c-40d5-99f9-5b52a6f49710-plugins-config-volume" pod "70e72a12-774c-40d5-99f9-5b52a6f49710" (UID: "70e72a12-774c-40d5-99f9-5b52a6f49710"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:06.914154 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:06.914128    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e72a12-774c-40d5-99f9-5b52a6f49710-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:06.915748 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:06.915724    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e72a12-774c-40d5-99f9-5b52a6f49710-kube-api-access-c6v9v" pod "70e72a12-774c-40d5-99f9-5b52a6f49710" (UID: "70e72a12-774c-40d5-99f9-5b52a6f49710"). InnerVolumeSpecName "kube-api-access-c6v9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:06.953218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:06.953185    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:06.984732 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:52:06.984696    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:52:07.014547 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.014511    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/985cb344-e903-473b-b612-25eeebb99553-kube-api-access-2mrcj\" (UniqueName: \"kubernetes.io/projected/985cb344-e903-473b-b612-25eeebb99553-kube-api-access-2mrcj\") pod \"985cb344-e903-473b-b612-25eeebb99553\" (UID: \"985cb344-e903-473b-b612-25eeebb99553\") "
Oct 31 03:52:07.014707 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.014609    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/985cb344-e903-473b-b612-25eeebb99553-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985cb344-e903-473b-b612-25eeebb99553-plugins-config-volume\") pod \"985cb344-e903-473b-b612-25eeebb99553\" (UID: \"985cb344-e903-473b-b612-25eeebb99553\") "
Oct 31 03:52:07.015003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.014981    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985cb344-e903-473b-b612-25eeebb99553-plugins-config-volume" pod "985cb344-e903-473b-b612-25eeebb99553" (UID: "985cb344-e903-473b-b612-25eeebb99553"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:07.015216 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.015199    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/985cb344-e903-473b-b612-25eeebb99553-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:07.015277 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.015217    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6v9v\" (UniqueName: \"kubernetes.io/projected/70e72a12-774c-40d5-99f9-5b52a6f49710-kube-api-access-c6v9v\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:07.016609 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.016592    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985cb344-e903-473b-b612-25eeebb99553-kube-api-access-2mrcj" pod "985cb344-e903-473b-b612-25eeebb99553" (UID: "985cb344-e903-473b-b612-25eeebb99553"). InnerVolumeSpecName "kube-api-access-2mrcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:07.122907 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.122806    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mrcj\" (UniqueName: \"kubernetes.io/projected/985cb344-e903-473b-b612-25eeebb99553-kube-api-access-2mrcj\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:07.415152 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.415069    4900 generic.go:356] "Generic (PLEG): container finished" podID="985cb344-e903-473b-b612-25eeebb99553" containerID="aba88c9bbab6b4ebaff0d80246350e045e7471631e2ab935eabbbcf4acf57530" exitCode=0
Oct 31 03:52:07.415152 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.415116    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"
Oct 31 03:52:07.415364 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.415154    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p" event={"ID":"985cb344-e903-473b-b612-25eeebb99553","Type":"ContainerDied","Data":"aba88c9bbab6b4ebaff0d80246350e045e7471631e2ab935eabbbcf4acf57530"}
Oct 31 03:52:07.415364 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.415210    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p" event={"ID":"985cb344-e903-473b-b612-25eeebb99553","Type":"ContainerDied","Data":"33a3b75485b6efa1389cef7b7848f7e0d2bd00bd9bb725afe7e0cf50d045eeb3"}
Oct 31 03:52:07.415364 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.415239    4900 scope.go:122] "RemoveContainer" containerID="aba88c9bbab6b4ebaff0d80246350e045e7471631e2ab935eabbbcf4acf57530"
Oct 31 03:52:07.416559 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.416535    4900 generic.go:356] "Generic (PLEG): container finished" podID="70e72a12-774c-40d5-99f9-5b52a6f49710" containerID="89e75987e1457c3546ad5611d661f21d4ef83a90d7ec19d92a92940d62f549fa" exitCode=0
Oct 31 03:52:07.416687 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.416566    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"
Oct 31 03:52:07.416687 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.416616    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z" event={"ID":"70e72a12-774c-40d5-99f9-5b52a6f49710","Type":"ContainerDied","Data":"89e75987e1457c3546ad5611d661f21d4ef83a90d7ec19d92a92940d62f549fa"}
Oct 31 03:52:07.416687 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.416665    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z" event={"ID":"70e72a12-774c-40d5-99f9-5b52a6f49710","Type":"ContainerDied","Data":"08f2d732dbed9ad0fe8ad81a15bcc2a2317686eb8796aeef9fdf0b4d47cc64fb"}
Oct 31 03:52:07.424941 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.424915    4900 scope.go:122] "RemoveContainer" containerID="aba88c9bbab6b4ebaff0d80246350e045e7471631e2ab935eabbbcf4acf57530"
Oct 31 03:52:07.425214 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:52:07.425193    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba88c9bbab6b4ebaff0d80246350e045e7471631e2ab935eabbbcf4acf57530\": container with ID starting with aba88c9bbab6b4ebaff0d80246350e045e7471631e2ab935eabbbcf4acf57530 not found: ID does not exist" containerID="aba88c9bbab6b4ebaff0d80246350e045e7471631e2ab935eabbbcf4acf57530"
Oct 31 03:52:07.425284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.425222    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba88c9bbab6b4ebaff0d80246350e045e7471631e2ab935eabbbcf4acf57530"} err="failed to get container status \"aba88c9bbab6b4ebaff0d80246350e045e7471631e2ab935eabbbcf4acf57530\": rpc error: code = NotFound desc = could not find container \"aba88c9bbab6b4ebaff0d80246350e045e7471631e2ab935eabbbcf4acf57530\": container with ID starting with aba88c9bbab6b4ebaff0d80246350e045e7471631e2ab935eabbbcf4acf57530 not found: ID does not exist"
Oct 31 03:52:07.425284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.425239    4900 scope.go:122] "RemoveContainer" containerID="89e75987e1457c3546ad5611d661f21d4ef83a90d7ec19d92a92940d62f549fa"
Oct 31 03:52:07.433409 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.433390    4900 scope.go:122] "RemoveContainer" containerID="89e75987e1457c3546ad5611d661f21d4ef83a90d7ec19d92a92940d62f549fa"
Oct 31 03:52:07.433675 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:52:07.433657    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89e75987e1457c3546ad5611d661f21d4ef83a90d7ec19d92a92940d62f549fa\": container with ID starting with 89e75987e1457c3546ad5611d661f21d4ef83a90d7ec19d92a92940d62f549fa not found: ID does not exist" containerID="89e75987e1457c3546ad5611d661f21d4ef83a90d7ec19d92a92940d62f549fa"
Oct 31 03:52:07.433734 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.433681    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89e75987e1457c3546ad5611d661f21d4ef83a90d7ec19d92a92940d62f549fa"} err="failed to get container status \"89e75987e1457c3546ad5611d661f21d4ef83a90d7ec19d92a92940d62f549fa\": rpc error: code = NotFound desc = could not find container \"89e75987e1457c3546ad5611d661f21d4ef83a90d7ec19d92a92940d62f549fa\": container with ID starting with 89e75987e1457c3546ad5611d661f21d4ef83a90d7ec19d92a92940d62f549fa not found: ID does not exist"
Oct 31 03:52:07.580589 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.580550    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"]
Oct 31 03:52:07.585579 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.585551    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca15df-5d7b6466bc-7sr7p"]
Oct 31 03:52:07.695657 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.695567    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"]
Oct 31 03:52:07.700257 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:07.700233    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca15df-598f689fdd-dt47z"]
Oct 31 03:52:08.284971 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:08.284931    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:09.153669 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:09.153634    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70e72a12-774c-40d5-99f9-5b52a6f49710" path="/var/lib/kubelet/pods/70e72a12-774c-40d5-99f9-5b52a6f49710/volumes"
Oct 31 03:52:09.154092 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:09.154082    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="985cb344-e903-473b-b612-25eeebb99553" path="/var/lib/kubelet/pods/985cb344-e903-473b-b612-25eeebb99553/volumes"
Oct 31 03:52:09.974345 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:09.974314    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:10.185969 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:10.185939    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:11.648088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:11.648055    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:12.230321 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:12.230287    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:13.141891 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:13.141861    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:13.440114 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:13.440022    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:13.924525 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:13.924493    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:14.660088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:14.660053    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:14.916660 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:14.916578    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:18.931629 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:18.931593    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:19.228178 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:19.228081    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:22.233690 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:22.233654    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:22.502482 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:22.502388    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:23.722114 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:23.722078    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:25.604714 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:25.604681    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:27.634223 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:27.634182    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:27.846371 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:27.846342    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:28.545607 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:28.545528    4900 generic.go:356] "Generic (PLEG): container finished" podID="fba5a14c-8608-4005-961d-869f98e5d3a7" containerID="a1994aebfe40728316c650405921475a95188999833b35fddc88feb3a6451a16" exitCode=0
Oct 31 03:52:28.545802 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:28.545604    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2" event={"ID":"fba5a14c-8608-4005-961d-869f98e5d3a7","Type":"ContainerDied","Data":"a1994aebfe40728316c650405921475a95188999833b35fddc88feb3a6451a16"}
Oct 31 03:52:28.546100 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:28.546085    4900 scope.go:122] "RemoveContainer" containerID="a1994aebfe40728316c650405921475a95188999833b35fddc88feb3a6451a16"
Oct 31 03:52:28.547572 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:28.547545    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:52:28.547659 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:28.547609    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:52:28.547659 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:28.547640    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:52:28.681578 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:28.681549    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:52:28.703390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:28.703359    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:52:29.275466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:29.275431    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:29.565874 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:29.565760    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2" event={"ID":"fba5a14c-8608-4005-961d-869f98e5d3a7","Type":"ContainerStarted","Data":"3bc3cfa2d96cd57f1b29679ad196738b04f6c87c3d8ef6edf9df869fbfd8a875"}
Oct 31 03:52:29.824767 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:29.824680    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:29.831629 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:29.831606    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:52:29.831863 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:29.831839    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 03:52:29.831929 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:29.831895    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2" podUID="fba5a14c-8608-4005-961d-869f98e5d3a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 03:52:30.697859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:30.697781    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:30.832086 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:30.832053    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 03:52:30.832497 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:30.832113    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2" podUID="fba5a14c-8608-4005-961d-869f98e5d3a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 03:52:30.942103 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:30.942065    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:31.831950 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:31.831906    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 03:52:31.832143 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:31.831982    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2" podUID="fba5a14c-8608-4005-961d-869f98e5d3a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 03:52:32.832209 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:32.832171    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 03:52:32.832616 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:32.832230    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2" podUID="fba5a14c-8608-4005-961d-869f98e5d3a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 03:52:33.831080 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:33.831047    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 03:52:33.831276 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:33.831117    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2" podUID="fba5a14c-8608-4005-961d-869f98e5d3a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 03:52:34.814781 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:34.814736    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:34.836360 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:34.836327    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:52:34.836746 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:34.836714    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:52:34.837446 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:34.837426    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 03:52:35.336918 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:35.336885    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:36.437348 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:36.437313    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:36.834993 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:36.834962    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:36.986178 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:52:36.986142    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:52:37.045102 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:37.045042    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:37.268587 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:37.268554    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:37.790890 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:37.790861    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:38.345841 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:38.345801    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:38.552631 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:38.552599    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:41.103326 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:41.103239    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:41.327133 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:41.327100    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:41.580810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:41.572813    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:41.816400 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:41.816363    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:42.246730 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:42.246698    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:43.216926 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:43.216887    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:43.419132 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:43.419100    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:43.921590 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:43.921552    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:44.797431 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:44.797392    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:45.003280 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:45.003241    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:47.292029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:47.291991    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:47.525069 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:47.525034    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:48.689004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:48.688968    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:49.781346 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:49.781315    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:49.995060 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:49.995019    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:50.573478 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:50.573446    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:50.796120 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:50.796090    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:51.773649 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:51.773614    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:52.189459 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:52.189377    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"]
Oct 31 03:52:52.189882 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:52.189603    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl" podUID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" containerName="istio-proxy" containerID="cri-o://bfe0122fd5106a6e3cda160d76462e5e26a366032bf79123a7358e69afc71649" gracePeriod=30
Oct 31 03:52:52.270433 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:52.270391    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:54.435388 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:54.435356    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:57.446047 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.446023    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:52:57.512120 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.512093    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:57.552000 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.551977    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-workload-certs\") pod \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") "
Oct 31 03:52:57.552116 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.552046    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-envoy\") pod \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") "
Oct 31 03:52:57.552167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.552115    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-podinfo\") pod \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") "
Oct 31 03:52:57.552222 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.552163    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-credential-socket\") pod \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") "
Oct 31 03:52:57.552288 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.552212    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-workload-certs" pod "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" (UID: "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:57.552288 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.552246    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istiod-ca-cert\") pod \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") "
Oct 31 03:52:57.552398 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.552328    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-workload-socket\") pod \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") "
Oct 31 03:52:57.552462 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.552395    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-data\") pod \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") "
Oct 31 03:52:57.552462 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.552428    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-token\" (UniqueName: \"kubernetes.io/projected/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-token\") pod \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") "
Oct 31 03:52:57.552462 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.552435    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-credential-socket" pod "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" (UID: "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:57.552462 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.552456    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-kube-api-access-mbhcp\" (UniqueName: \"kubernetes.io/projected/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-kube-api-access-mbhcp\") pod \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\" (UID: \"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9\") "
Oct 31 03:52:57.552711 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.552642    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-data" pod "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" (UID: "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:57.552711 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.552655    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istiod-ca-cert" pod "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" (UID: "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:57.552909 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.552887    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-workload-socket" pod "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" (UID: "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:57.553183 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.553157    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.553269 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.553189    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.553269 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.553205    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.553269 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.553221    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.553269 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.553237    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.554378 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.554349    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-envoy" pod "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" (UID: "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:57.554480 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.554417    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-token" pod "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" (UID: "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:57.554480 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.554447    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-podinfo" pod "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" (UID: "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:52:57.554596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.554566    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-kube-api-access-mbhcp" pod "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" (UID: "9ca7d913-ef17-4f3a-8e07-bab6ba2048c9"). InnerVolumeSpecName "kube-api-access-mbhcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:57.654876 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.654849    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.654876 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.654873    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.655033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.654882    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.655033 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.654890    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbhcp\" (UniqueName: \"kubernetes.io/projected/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9-kube-api-access-mbhcp\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:57.726297 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.726268    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:57.739232 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.739203    4900 generic.go:356] "Generic (PLEG): container finished" podID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" containerID="bfe0122fd5106a6e3cda160d76462e5e26a366032bf79123a7358e69afc71649" exitCode=0
Oct 31 03:52:57.739362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.739253    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"
Oct 31 03:52:57.739362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.739284    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl" event={"ID":"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9","Type":"ContainerDied","Data":"bfe0122fd5106a6e3cda160d76462e5e26a366032bf79123a7358e69afc71649"}
Oct 31 03:52:57.739362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.739336    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl" event={"ID":"9ca7d913-ef17-4f3a-8e07-bab6ba2048c9","Type":"ContainerDied","Data":"3e3e098f96eeb06b21784e5935241aa548c394b87ab7fa02f8a5c82e4ff93048"}
Oct 31 03:52:57.739492 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.739362    4900 scope.go:122] "RemoveContainer" containerID="bfe0122fd5106a6e3cda160d76462e5e26a366032bf79123a7358e69afc71649"
Oct 31 03:52:57.752391 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.752372    4900 scope.go:122] "RemoveContainer" containerID="b012409c9a55b409ccbfcfbb322d6ec29370694a9e41a37ae541c8ec5f3feb67"
Oct 31 03:52:57.761243 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.761216    4900 scope.go:122] "RemoveContainer" containerID="bfe0122fd5106a6e3cda160d76462e5e26a366032bf79123a7358e69afc71649"
Oct 31 03:52:57.761540 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:52:57.761515    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe0122fd5106a6e3cda160d76462e5e26a366032bf79123a7358e69afc71649\": container with ID starting with bfe0122fd5106a6e3cda160d76462e5e26a366032bf79123a7358e69afc71649 not found: ID does not exist" containerID="bfe0122fd5106a6e3cda160d76462e5e26a366032bf79123a7358e69afc71649"
Oct 31 03:52:57.761616 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.761546    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe0122fd5106a6e3cda160d76462e5e26a366032bf79123a7358e69afc71649"} err="failed to get container status \"bfe0122fd5106a6e3cda160d76462e5e26a366032bf79123a7358e69afc71649\": rpc error: code = NotFound desc = could not find container \"bfe0122fd5106a6e3cda160d76462e5e26a366032bf79123a7358e69afc71649\": container with ID starting with bfe0122fd5106a6e3cda160d76462e5e26a366032bf79123a7358e69afc71649 not found: ID does not exist"
Oct 31 03:52:57.761616 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.761565    4900 scope.go:122] "RemoveContainer" containerID="b012409c9a55b409ccbfcfbb322d6ec29370694a9e41a37ae541c8ec5f3feb67"
Oct 31 03:52:57.761856 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:52:57.761770    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b012409c9a55b409ccbfcfbb322d6ec29370694a9e41a37ae541c8ec5f3feb67\": container with ID starting with b012409c9a55b409ccbfcfbb322d6ec29370694a9e41a37ae541c8ec5f3feb67 not found: ID does not exist" containerID="b012409c9a55b409ccbfcfbb322d6ec29370694a9e41a37ae541c8ec5f3feb67"
Oct 31 03:52:57.761856 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.761828    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b012409c9a55b409ccbfcfbb322d6ec29370694a9e41a37ae541c8ec5f3feb67"} err="failed to get container status \"b012409c9a55b409ccbfcfbb322d6ec29370694a9e41a37ae541c8ec5f3feb67\": rpc error: code = NotFound desc = could not find container \"b012409c9a55b409ccbfcfbb322d6ec29370694a9e41a37ae541c8ec5f3feb67\": container with ID starting with b012409c9a55b409ccbfcfbb322d6ec29370694a9e41a37ae541c8ec5f3feb67 not found: ID does not exist"
Oct 31 03:52:57.905731 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.905659    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"]
Oct 31 03:52:57.909135 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:57.909117    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-e2f0bd-openshift-default-746bc78f85-d7xtl"]
Oct 31 03:52:58.853957 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:58.853921    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:59.152967 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:59.152892    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" path="/var/lib/kubelet/pods/9ca7d913-ef17-4f3a-8e07-bab6ba2048c9/volumes"
Oct 31 03:52:59.314681 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:59.314650    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:59.876090 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:52:59.876058    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:00.095083 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:00.095051    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:00.338825 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:00.338780    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:04.698532 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:04.698482    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:04.942543 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:04.942511    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:05.345485 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:05.345448    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:06.986436 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:53:06.986391    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:53:07.998490 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:07.998450    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:08.208247 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.208212    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:08.485653 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.485617    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"]
Oct 31 03:53:08.486762 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.486730    4900 cpu_manager.go:414] "containerMap: removing container" podUID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" containerName="istio-proxy"
Oct 31 03:53:08.486762 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.486761    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" containerName="istio-proxy"
Oct 31 03:53:08.486978 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.486806    4900 cpu_manager.go:414] "containerMap: removing container" podUID="70e72a12-774c-40d5-99f9-5b52a6f49710" containerName="payload-processing"
Oct 31 03:53:08.486978 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.486815    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70e72a12-774c-40d5-99f9-5b52a6f49710" containerName="payload-processing"
Oct 31 03:53:08.486978 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.486874    4900 cpu_manager.go:414] "containerMap: removing container" podUID="985cb344-e903-473b-b612-25eeebb99553" containerName="payload-pre-processing"
Oct 31 03:53:08.486978 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.486883    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985cb344-e903-473b-b612-25eeebb99553" containerName="payload-pre-processing"
Oct 31 03:53:08.487218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.487020    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" containerName="istio-proxy"
Oct 31 03:53:08.487218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.487036    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="70e72a12-774c-40d5-99f9-5b52a6f49710" containerName="payload-processing"
Oct 31 03:53:08.487218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.487046    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" containerName="istio-proxy"
Oct 31 03:53:08.487218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.487062    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="985cb344-e903-473b-b612-25eeebb99553" containerName="payload-pre-processing"
Oct 31 03:53:08.492767 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.492743    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"
Oct 31 03:53:08.544902 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.544866    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"]
Oct 31 03:53:08.590564 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.590516    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-a5f70c\""
Oct 31 03:53:08.590900 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.590605    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-a5f70c-dockercfg-wfs9f\""
Oct 31 03:53:08.592826 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.592428    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9a50ed-4995-42b7-b5f9-76dead1711c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62\" (UID: \"4c9a50ed-4995-42b7-b5f9-76dead1711c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"
Oct 31 03:53:08.592826 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.592713    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xsqz\" (UniqueName: \"kubernetes.io/projected/4c9a50ed-4995-42b7-b5f9-76dead1711c7-kube-api-access-2xsqz\") pod \"payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62\" (UID: \"4c9a50ed-4995-42b7-b5f9-76dead1711c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"
Oct 31 03:53:08.693796 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.693763    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xsqz\" (UniqueName: \"kubernetes.io/projected/4c9a50ed-4995-42b7-b5f9-76dead1711c7-kube-api-access-2xsqz\") pod \"payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62\" (UID: \"4c9a50ed-4995-42b7-b5f9-76dead1711c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"
Oct 31 03:53:08.693991 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.693870    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9a50ed-4995-42b7-b5f9-76dead1711c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62\" (UID: \"4c9a50ed-4995-42b7-b5f9-76dead1711c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"
Oct 31 03:53:08.694583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.694563    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9a50ed-4995-42b7-b5f9-76dead1711c7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62\" (UID: \"4c9a50ed-4995-42b7-b5f9-76dead1711c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"
Oct 31 03:53:08.751433 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.751367    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xsqz\" (UniqueName: \"kubernetes.io/projected/4c9a50ed-4995-42b7-b5f9-76dead1711c7-kube-api-access-2xsqz\") pod \"payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62\" (UID: \"4c9a50ed-4995-42b7-b5f9-76dead1711c7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"
Oct 31 03:53:08.812579 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:08.812545    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"
Oct 31 03:53:09.184320 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:53:09.184291    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9a50ed_4995_42b7_b5f9_76dead1711c7.slice/crio-0bab9d654f6c9a584fe187c6a9ddbdfb851bb434251ce8d892809f1c6fe80a06 WatchSource:0}: Error finding container 0bab9d654f6c9a584fe187c6a9ddbdfb851bb434251ce8d892809f1c6fe80a06: Status 404 returned error can't find the container with id 0bab9d654f6c9a584fe187c6a9ddbdfb851bb434251ce8d892809f1c6fe80a06
Oct 31 03:53:09.187086 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:09.187063    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"]
Oct 31 03:53:09.813030 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:09.812990    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62" event={"ID":"4c9a50ed-4995-42b7-b5f9-76dead1711c7","Type":"ContainerStarted","Data":"bb035c7bfb44ce61a937def286ab378764cd5cde775514005104ac829ae19640"}
Oct 31 03:53:09.813414 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:09.813048    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62" event={"ID":"4c9a50ed-4995-42b7-b5f9-76dead1711c7","Type":"ContainerStarted","Data":"0bab9d654f6c9a584fe187c6a9ddbdfb851bb434251ce8d892809f1c6fe80a06"}
Oct 31 03:53:09.813414 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:09.813127    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"
Oct 31 03:53:09.936711 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:09.936655    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62" podStartSLOduration=1.936641414 podStartE2EDuration="1.936641414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:09.935544612 +0000 UTC m=+2742.901877955" watchObservedRunningTime="2026-10-31 03:53:09.936641414 +0000 UTC m=+2742.902974757"
Oct 31 03:53:10.823982 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:10.823917    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:11.045367 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:11.045332    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:11.305247 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:11.305215    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:12.544137 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.544101    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"]
Oct 31 03:53:12.544876 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.544853    4900 cpu_manager.go:414] "containerMap: removing container" podUID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" containerName="istio-proxy"
Oct 31 03:53:12.544876 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.544876    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ca7d913-ef17-4f3a-8e07-bab6ba2048c9" containerName="istio-proxy"
Oct 31 03:53:12.548567 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.548547    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.600990 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.600966    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 03:53:12.600990 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.600981    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-r7bff\""
Oct 31 03:53:12.601196 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.600966    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 03:53:12.601196 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.600966    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 03:53:12.643892 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.643862    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.644043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.643975    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.644043 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.644012    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.644157 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.644126    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.644210 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.644170    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.644310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.644294    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcfqh\" (UniqueName: \"kubernetes.io/projected/8e1017c9-1f08-4441-adfd-90415b9c6f17-kube-api-access-hcfqh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.644362 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.644329    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1017c9-1f08-4441-adfd-90415b9c6f17-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.654290 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.654267    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"]
Oct 31 03:53:12.745296 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.745266    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcfqh\" (UniqueName: \"kubernetes.io/projected/8e1017c9-1f08-4441-adfd-90415b9c6f17-kube-api-access-hcfqh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.745475 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.745304    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1017c9-1f08-4441-adfd-90415b9c6f17-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.745475 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.745381    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.745475 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.745435    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.745475 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.745468    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.745765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.745567    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.745765 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.745619    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.746416 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.746390    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.747902 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.747883    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.747902 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.747890    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1017c9-1f08-4441-adfd-90415b9c6f17-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.748036 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.747892    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.748036 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.747966    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.803189 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.803128    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.803332 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.803312    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcfqh\" (UniqueName: \"kubernetes.io/projected/8e1017c9-1f08-4441-adfd-90415b9c6f17-kube-api-access-hcfqh\") pod \"istiod-openshift-gateway-7ddd9cfdc8-cmg7d\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:12.861203 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:12.861174    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:13.296936 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:53:13.296905    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e1017c9_1f08_4441_adfd_90415b9c6f17.slice/crio-14e9a83d6243af7b83bb82b85d9909dd3c95b74863a7f1ee6b413a2867d4c4ca WatchSource:0}: Error finding container 14e9a83d6243af7b83bb82b85d9909dd3c95b74863a7f1ee6b413a2867d4c4ca: Status 404 returned error can't find the container with id 14e9a83d6243af7b83bb82b85d9909dd3c95b74863a7f1ee6b413a2867d4c4ca
Oct 31 03:53:13.298499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:13.298470    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"]
Oct 31 03:53:13.299544 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:13.299512    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:53:13.299647 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:13.299570    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:53:13.839152 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:13.839119    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d" event={"ID":"8e1017c9-1f08-4441-adfd-90415b9c6f17","Type":"ContainerStarted","Data":"98c64a37313e85ccde311649ccce5838ee4c239ddee757ba3d54c005aedfb07f"}
Oct 31 03:53:13.839647 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:13.839174    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d" event={"ID":"8e1017c9-1f08-4441-adfd-90415b9c6f17","Type":"ContainerStarted","Data":"14e9a83d6243af7b83bb82b85d9909dd3c95b74863a7f1ee6b413a2867d4c4ca"}
Oct 31 03:53:13.839647 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:13.839238    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:13.958178 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:13.958123    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d" podStartSLOduration=1.958111019 podStartE2EDuration="1.958111019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:13.956700603 +0000 UTC m=+2746.923033941" watchObservedRunningTime="2026-10-31 03:53:13.958111019 +0000 UTC m=+2746.924444364"
Oct 31 03:53:14.847700 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:14.847667    4900 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-cmg7d container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:53:14.848092 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:14.847727    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d" podUID="8e1017c9-1f08-4441-adfd-90415b9c6f17" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:53:16.692066 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:16.692032    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:17.846681 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:17.846647    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:53:20.822733 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:20.822702    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"
Oct 31 03:53:21.316233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:21.316193    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:26.269613 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:26.269584    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:28.997896 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:28.997862    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:29.659757 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:29.659718    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:29.885237 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:29.885199    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:31.186466 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:31.186431    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:33.975176 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:33.975137    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:34.195211 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:34.195179    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:35.704429 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.704396    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"]
Oct 31 03:53:35.708565 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.708546    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.804970 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.804935    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"]
Oct 31 03:53:35.838993 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.838950    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.839204 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.839177    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.839255 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.839231    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.839355 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.839336    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.839402 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.839378    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhmbs\" (UniqueName: \"kubernetes.io/projected/73481f65-a2ac-45c1-be4e-eb29aa0915a9-kube-api-access-dhmbs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.839463 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.839404    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.839522 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.839461    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.839522 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.839512    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.839619 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.839566    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.851776 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.851755    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-dd5b85-openshift-default-dockercfg-b24df\""
Oct 31 03:53:35.864585 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.864560    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:35.940497 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.940461    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.940497 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.940499    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.940766 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.940550    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.940766 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.940634    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.940766 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.940654    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.940766 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.940744    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.941024 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.940805    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmbs\" (UniqueName: \"kubernetes.io/projected/73481f65-a2ac-45c1-be4e-eb29aa0915a9-kube-api-access-dhmbs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.941024 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.940843    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.941024 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.940865    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.941190 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.941105    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.941190 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.941177    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.941294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.941196    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.941496 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.941468    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.941591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.941541    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.943079 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.943056    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.943185 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.943165    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.997709 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.997679    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:35.997914 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:35.997891    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmbs\" (UniqueName: \"kubernetes.io/projected/73481f65-a2ac-45c1-be4e-eb29aa0915a9-kube-api-access-dhmbs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:36.022953 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:36.022929    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:36.316746 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:36.316649    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:36.441212 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:53:36.441173    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73481f65_a2ac_45c1_be4e_eb29aa0915a9.slice/crio-2d08dc31b406c5939dbd3148ad4bc7f317ad1fdcd4169d018ac71f816790cdc7 WatchSource:0}: Error finding container 2d08dc31b406c5939dbd3148ad4bc7f317ad1fdcd4169d018ac71f816790cdc7: Status 404 returned error can't find the container with id 2d08dc31b406c5939dbd3148ad4bc7f317ad1fdcd4169d018ac71f816790cdc7
Oct 31 03:53:36.444307 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:36.444268    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:53:36.444416 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:36.444339    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:53:36.444416 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:36.444369    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:53:36.444497 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:36.444469    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"]
Oct 31 03:53:36.527101 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:36.527067    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:36.975544 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:36.975502    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" event={"ID":"73481f65-a2ac-45c1-be4e-eb29aa0915a9","Type":"ContainerStarted","Data":"ea601a4aa2c83cd331c7186d88c654921acdc374ae199454f1114902633dd41b"}
Oct 31 03:53:36.976065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:36.975565    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" event={"ID":"73481f65-a2ac-45c1-be4e-eb29aa0915a9","Type":"ContainerStarted","Data":"2d08dc31b406c5939dbd3148ad4bc7f317ad1fdcd4169d018ac71f816790cdc7"}
Oct 31 03:53:36.986193 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:53:36.986161    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:53:37.023137 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:37.023103    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:37.024797 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:37.024758    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:53:37.024915 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:37.024824    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" podUID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 03:53:37.095856 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:37.095778    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" podStartSLOduration=2.095764782 podStartE2EDuration="2.095764782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:53:37.092918404 +0000 UTC m=+2770.059251747" watchObservedRunningTime="2026-10-31 03:53:37.095764782 +0000 UTC m=+2770.062098134"
Oct 31 03:53:38.023813 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:38.023755    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:53:38.024201 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:38.023830    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" podUID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 03:53:39.023620 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:39.023585    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:53:39.023822 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:39.023639    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" podUID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 03:53:39.607921 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:39.607887    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:39.826459 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:39.826418    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:40.087500 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:40.087463    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:53:40.094625 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:40.089772    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" podUID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 03:53:41.024302 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:41.024267    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:53:41.024845 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:41.024333    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" podUID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 03:53:42.024171 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:42.024132    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:53:42.024358 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:42.024185    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" podUID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 03:53:42.080808 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:42.080192    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:43.023593 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:43.023558    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:53:43.023774 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:43.023615    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" podUID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 03:53:44.024073 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:44.024034    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 03:53:44.024491 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:44.024100    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" podUID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 03:53:46.063964 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:46.063920    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:53:46.064486 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:46.063977    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" podUID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:53:46.088370 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:46.086988    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:46.088370 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:46.087431    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:46.088685 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:46.088654    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:53:47.260597 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:47.260566    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:47.469937 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:47.469908    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:51.488408 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:51.488372    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:51.696603 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:51.696567    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:52.374175 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:52.374082    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:57.437264 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:57.437219    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:57.905731 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:53:57.905692    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:02.463099 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:02.463028    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:06.985142 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:54:06.985101    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:54:07.456400 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:07.456314    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:07.659921 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:07.659891    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:07.886861 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:07.886832    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:12.837251 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:12.837174    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:17.781112 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:17.781070    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:23.164591 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:23.164559    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:26.827535 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:26.827503    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:28.168426 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:28.168373    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:28.618102 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:28.618070    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:28.848081 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:28.848047    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:29.101502 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:29.101460    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:33.124101 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:33.124067    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:33.519620 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:33.519590    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:33.977039 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:33.976957    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:36.568840 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:36.568808    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:36.794933 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:36.794901    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:37.084022 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:54:37.083985    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:54:39.702560 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:39.702507    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:44.943126 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:44.943091    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:45.165922 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:45.165844    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:45.371638 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:45.371595    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:50.741058 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:50.741023    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:55.958811 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:55.958747    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:58.254149 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:58.254111    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:58.498403 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:58.498371    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:54:59.503050 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:54:59.503019    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:01.600054 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:01.600016    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:04.771188 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:04.771144    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:06.963125 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:06.963094    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:06.987335 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:55:06.987297    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:55:10.748029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:10.747992    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:12.232718 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:12.232648    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:13.217365 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:13.217329    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:17.221285 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:17.221253    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:22.092142 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:22.092104    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:27.209969 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:27.209930    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:28.212090 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:28.212054    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:32.899357 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:32.899326    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:36.985045 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:55:36.985009    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:55:37.905134 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:37.905100    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:42.892029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:42.891490    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:44.328722 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.328685    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"]
Oct 31 03:55:44.333110 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.333087    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.390120 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.390092    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrctp\" (UniqueName: \"kubernetes.io/projected/97819a0a-0328-4845-b5ca-c25af6385a11-kube-api-access-xrctp\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.390242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.390139    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-credential-socket\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.390242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.390208    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97819a0a-0328-4845-b5ca-c25af6385a11-istiod-ca-cert\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.390321 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.390279    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-istio-data\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.390361 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.390322    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97819a0a-0328-4845-b5ca-c25af6385a11-istio-podinfo\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.390361 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.390342    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-istio-envoy\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.390425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.390412    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97819a0a-0328-4845-b5ca-c25af6385a11-istio-token\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.390536 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.390523    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-workload-certs\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.390581 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.390569    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-workload-socket\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.435465 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.435426    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-2339ed-openshift-default-dockercfg-q2fqj\""
Oct 31 03:55:44.438525 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.438501    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"]
Oct 31 03:55:44.492217 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.492192    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-workload-certs\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.492345 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.492244    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-workload-socket\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.492345 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.492301    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrctp\" (UniqueName: \"kubernetes.io/projected/97819a0a-0328-4845-b5ca-c25af6385a11-kube-api-access-xrctp\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.492345 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.492332    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-credential-socket\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.492499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.492351    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97819a0a-0328-4845-b5ca-c25af6385a11-istiod-ca-cert\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.492499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.492388    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-istio-data\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.492499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.492420    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97819a0a-0328-4845-b5ca-c25af6385a11-istio-podinfo\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.492499 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.492445    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-istio-envoy\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.492816 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.492626    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-workload-certs\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.492816 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.492623    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97819a0a-0328-4845-b5ca-c25af6385a11-istio-token\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.492943 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.492885    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-istio-data\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.492943 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.492917    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-workload-socket\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.493027 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.493009    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-credential-socket\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.493312 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.493295    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97819a0a-0328-4845-b5ca-c25af6385a11-istiod-ca-cert\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.494640 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.494620    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-istio-envoy\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.494927 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.494906    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97819a0a-0328-4845-b5ca-c25af6385a11-istio-podinfo\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.549142 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.549114    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97819a0a-0328-4845-b5ca-c25af6385a11-istio-token\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.549243 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.549117    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrctp\" (UniqueName: \"kubernetes.io/projected/97819a0a-0328-4845-b5ca-c25af6385a11-kube-api-access-xrctp\") pod \"e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") " pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:44.648476 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:44.648398    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:45.066841 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:55:45.066774    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97819a0a_0328_4845_b5ca_c25af6385a11.slice/crio-286a2b856fa8c6e81bde0422645ab489e72990462e1d32a9666ff0da6c7ad6c7 WatchSource:0}: Error finding container 286a2b856fa8c6e81bde0422645ab489e72990462e1d32a9666ff0da6c7ad6c7: Status 404 returned error can't find the container with id 286a2b856fa8c6e81bde0422645ab489e72990462e1d32a9666ff0da6c7ad6c7
Oct 31 03:55:45.068948 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:45.068920    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"]
Oct 31 03:55:45.069552 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:45.069522    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:55:45.069596 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:45.069586    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:55:45.069631 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:45.069617    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:55:45.760441 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:45.760394    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8" event={"ID":"97819a0a-0328-4845-b5ca-c25af6385a11","Type":"ContainerStarted","Data":"5e72147ab15ee0d2590fa42f6238dbbf274746ca75d3a0db8b9d5f07cc4e27d8"}
Oct 31 03:55:45.760953 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:45.760471    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8" event={"ID":"97819a0a-0328-4845-b5ca-c25af6385a11","Type":"ContainerStarted","Data":"286a2b856fa8c6e81bde0422645ab489e72990462e1d32a9666ff0da6c7ad6c7"}
Oct 31 03:55:45.885128 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:45.885061    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8" podStartSLOduration=1.885041848 podStartE2EDuration="1.885041848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:45.880743878 +0000 UTC m=+2898.847077225" watchObservedRunningTime="2026-10-31 03:55:45.885041848 +0000 UTC m=+2898.851375192"
Oct 31 03:55:46.656810 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:46.655876    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:46.660312 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:46.660271    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 03:55:46.660501 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:46.660327    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8" podUID="97819a0a-0328-4845-b5ca-c25af6385a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 03:55:47.658218 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:47.658176    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 03:55:47.658863 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:47.658828    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8" podUID="97819a0a-0328-4845-b5ca-c25af6385a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 03:55:47.686969 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:47.686931    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:47.911390 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:47.911300    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:48.116527 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:48.116495    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:48.655410 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:48.655373    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 03:55:48.655684 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:48.655657    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8" podUID="97819a0a-0328-4845-b5ca-c25af6385a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 03:55:49.652185 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:49.652143    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 03:55:49.652819 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:49.652770    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8" podUID="97819a0a-0328-4845-b5ca-c25af6385a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 03:55:50.653105 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:50.653067    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 03:55:50.653748 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:50.653702    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8" podUID="97819a0a-0328-4845-b5ca-c25af6385a11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 03:55:51.732969 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:51.732930    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:51.734223 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:51.734198    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:51.734802 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:51.734763    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 03:55:52.884317 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:52.884269    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:57.786009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:57.785978    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:55:58.609284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:55:58.609254    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:02.762238 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:02.762202    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:04.093520 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:04.093485    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:06.984968 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:56:06.984927    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:56:08.029132 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:08.029104    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:11.285224 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:11.284532    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:12.719993 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:12.719959    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:13.281970 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:13.281938    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:18.257312 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:18.257269    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:23.141285 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:23.141252    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:28.083541 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:28.083511    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:32.377320 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:32.377285    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"]
Oct 31 03:56:32.377717 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:32.377491    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn" podUID="b38b8385-a404-433d-bcee-5e5ffb33d5da" containerName="istio-proxy" containerID="cri-o://86761543486b7d4644fe8db9d0004575d2574f60a12446b38f9e990eca30e2e8" gracePeriod=30
Oct 31 03:56:32.434314 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:32.434281    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"]
Oct 31 03:56:32.434627 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:32.434610    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"]
Oct 31 03:56:32.435018 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:32.434989    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl" podUID="7092ff60-28f3-4737-897c-41020bc4a5a7" containerName="istio-proxy" containerID="cri-o://d4e19d0f8df7d8859832cbc3feaa55fc301c098976db8f75e663ca003b68e24a" gracePeriod=30
Oct 31 03:56:32.435436 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:32.435414    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd" podUID="44a6cc6d-73ae-42a5-899e-c8c05bf010f3" containerName="istio-proxy" containerID="cri-o://29918f8d23744016edd5b764b1b41f205240fd386218fa2686a0db6c182feda1" gracePeriod=30
Oct 31 03:56:32.477080 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:32.477045    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"]
Oct 31 03:56:32.477226 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:32.477202    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx" podUID="0dce71a3-b4c7-459d-92e8-c0515341abf1" containerName="istio-proxy" containerID="cri-o://ddf4554de2aadfe45045269273bff65d8ce203aac0bf23a945ca3f8932676eda" gracePeriod=30
Oct 31 03:56:32.986604 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:32.986572    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:36.984715 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:56:36.984670    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:56:37.701494 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.701467    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:56:37.758677 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.758644    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-credential-socket\") pod \"b38b8385-a404-433d-bcee-5e5ffb33d5da\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") "
Oct 31 03:56:37.758949 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.758711    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b38b8385-a404-433d-bcee-5e5ffb33d5da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38b8385-a404-433d-bcee-5e5ffb33d5da-istiod-ca-cert\") pod \"b38b8385-a404-433d-bcee-5e5ffb33d5da\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") "
Oct 31 03:56:37.758949 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.758745    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-data\") pod \"b38b8385-a404-433d-bcee-5e5ffb33d5da\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") "
Oct 31 03:56:37.759088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.758972    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-credential-socket" pod "b38b8385-a404-433d-bcee-5e5ffb33d5da" (UID: "b38b8385-a404-433d-bcee-5e5ffb33d5da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.759145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.759077    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-workload-certs\") pod \"b38b8385-a404-433d-bcee-5e5ffb33d5da\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") "
Oct 31 03:56:37.759145 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.759123    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-workload-socket\") pod \"b38b8385-a404-433d-bcee-5e5ffb33d5da\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") "
Oct 31 03:56:37.759253 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.759177    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38b8385-a404-433d-bcee-5e5ffb33d5da-kube-api-access-tgjmq\" (UniqueName: \"kubernetes.io/projected/b38b8385-a404-433d-bcee-5e5ffb33d5da-kube-api-access-tgjmq\") pod \"b38b8385-a404-433d-bcee-5e5ffb33d5da\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") "
Oct 31 03:56:37.759253 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.759224    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-data" pod "b38b8385-a404-433d-bcee-5e5ffb33d5da" (UID: "b38b8385-a404-433d-bcee-5e5ffb33d5da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.759365 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.759297    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-token\" (UniqueName: \"kubernetes.io/projected/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-token\") pod \"b38b8385-a404-433d-bcee-5e5ffb33d5da\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") "
Oct 31 03:56:37.759428 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.759368    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-envoy\") pod \"b38b8385-a404-433d-bcee-5e5ffb33d5da\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") "
Oct 31 03:56:37.759428 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.759405    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-workload-socket" pod "b38b8385-a404-433d-bcee-5e5ffb33d5da" (UID: "b38b8385-a404-433d-bcee-5e5ffb33d5da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.759428 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.759413    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-podinfo\") pod \"b38b8385-a404-433d-bcee-5e5ffb33d5da\" (UID: \"b38b8385-a404-433d-bcee-5e5ffb33d5da\") "
Oct 31 03:56:37.759612 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.759593    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38b8385-a404-433d-bcee-5e5ffb33d5da-istiod-ca-cert" pod "b38b8385-a404-433d-bcee-5e5ffb33d5da" (UID: "b38b8385-a404-433d-bcee-5e5ffb33d5da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:37.759722 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.759680    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-workload-certs" pod "b38b8385-a404-433d-bcee-5e5ffb33d5da" (UID: "b38b8385-a404-433d-bcee-5e5ffb33d5da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.760165 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.760139    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.760274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.760169    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38b8385-a404-433d-bcee-5e5ffb33d5da-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.760274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.760183    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.760274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.760197    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.760401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.760303    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.763199 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.763172    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-podinfo" pod "b38b8385-a404-433d-bcee-5e5ffb33d5da" (UID: "b38b8385-a404-433d-bcee-5e5ffb33d5da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:37.763310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.763195    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38b8385-a404-433d-bcee-5e5ffb33d5da-kube-api-access-tgjmq" pod "b38b8385-a404-433d-bcee-5e5ffb33d5da" (UID: "b38b8385-a404-433d-bcee-5e5ffb33d5da"). InnerVolumeSpecName "kube-api-access-tgjmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:37.763310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.763221    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-token" pod "b38b8385-a404-433d-bcee-5e5ffb33d5da" (UID: "b38b8385-a404-433d-bcee-5e5ffb33d5da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:37.764342 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.764319    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-envoy" pod "b38b8385-a404-433d-bcee-5e5ffb33d5da" (UID: "b38b8385-a404-433d-bcee-5e5ffb33d5da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.793932 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.793899    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:56:37.851673 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.851006    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:37.860747 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.860717    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-podinfo\") pod \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") "
Oct 31 03:56:37.860934 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.860773    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-workload-certs\") pod \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") "
Oct 31 03:56:37.860934 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.860812    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-credential-socket\") pod \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") "
Oct 31 03:56:37.860934 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.860842    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-kube-api-access-pm8rl\" (UniqueName: \"kubernetes.io/projected/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-kube-api-access-pm8rl\") pod \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") "
Oct 31 03:56:37.860934 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.860896    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-envoy\") pod \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") "
Oct 31 03:56:37.861184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.860937    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istiod-ca-cert\") pod \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") "
Oct 31 03:56:37.861184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.861028    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-workload-socket\") pod \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") "
Oct 31 03:56:37.861184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.861090    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-workload-certs" pod "44a6cc6d-73ae-42a5-899e-c8c05bf010f3" (UID: "44a6cc6d-73ae-42a5-899e-c8c05bf010f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.861184 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.861153    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-token\" (UniqueName: \"kubernetes.io/projected/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-token\") pod \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") "
Oct 31 03:56:37.861395 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.861236    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-data\") pod \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\" (UID: \"44a6cc6d-73ae-42a5-899e-c8c05bf010f3\") "
Oct 31 03:56:37.861607 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.861586    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-credential-socket" pod "44a6cc6d-73ae-42a5-899e-c8c05bf010f3" (UID: "44a6cc6d-73ae-42a5-899e-c8c05bf010f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.861954 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.861930    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.862065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.861961    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.862065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.861978    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.862065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.861995    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.862065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.862015    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgjmq\" (UniqueName: \"kubernetes.io/projected/b38b8385-a404-433d-bcee-5e5ffb33d5da-kube-api-access-tgjmq\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.862065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.862036    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38b8385-a404-433d-bcee-5e5ffb33d5da-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.862334 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.862092    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-workload-socket" pod "44a6cc6d-73ae-42a5-899e-c8c05bf010f3" (UID: "44a6cc6d-73ae-42a5-899e-c8c05bf010f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.862399 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.862332    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-data" pod "44a6cc6d-73ae-42a5-899e-c8c05bf010f3" (UID: "44a6cc6d-73ae-42a5-899e-c8c05bf010f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.862454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.862417    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istiod-ca-cert" pod "44a6cc6d-73ae-42a5-899e-c8c05bf010f3" (UID: "44a6cc6d-73ae-42a5-899e-c8c05bf010f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:37.863636 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.863614    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-kube-api-access-pm8rl" pod "44a6cc6d-73ae-42a5-899e-c8c05bf010f3" (UID: "44a6cc6d-73ae-42a5-899e-c8c05bf010f3"). InnerVolumeSpecName "kube-api-access-pm8rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:37.865195 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.865171    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-podinfo" pod "44a6cc6d-73ae-42a5-899e-c8c05bf010f3" (UID: "44a6cc6d-73ae-42a5-899e-c8c05bf010f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:37.865888 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.865863    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-token" pod "44a6cc6d-73ae-42a5-899e-c8c05bf010f3" (UID: "44a6cc6d-73ae-42a5-899e-c8c05bf010f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:37.866128 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.866103    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-envoy" pod "44a6cc6d-73ae-42a5-899e-c8c05bf010f3" (UID: "44a6cc6d-73ae-42a5-899e-c8c05bf010f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.867762 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.867739    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:56:37.871974 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.871953    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:56:37.963308 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963220    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-token\" (UniqueName: \"kubernetes.io/projected/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-token\") pod \"0dce71a3-b4c7-459d-92e8-c0515341abf1\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") "
Oct 31 03:56:37.963308 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963265    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dce71a3-b4c7-459d-92e8-c0515341abf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dce71a3-b4c7-459d-92e8-c0515341abf1-istiod-ca-cert\") pod \"0dce71a3-b4c7-459d-92e8-c0515341abf1\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") "
Oct 31 03:56:37.963308 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963294    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-workload-socket\") pod \"7092ff60-28f3-4737-897c-41020bc4a5a7\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") "
Oct 31 03:56:37.963624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963330    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-podinfo\") pod \"7092ff60-28f3-4737-897c-41020bc4a5a7\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") "
Oct 31 03:56:37.963624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963387    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-data\") pod \"0dce71a3-b4c7-459d-92e8-c0515341abf1\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") "
Oct 31 03:56:37.963624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963434    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-data\") pod \"7092ff60-28f3-4737-897c-41020bc4a5a7\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") "
Oct 31 03:56:37.963624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963463    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-envoy\") pod \"7092ff60-28f3-4737-897c-41020bc4a5a7\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") "
Oct 31 03:56:37.963624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963511    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-credential-socket\") pod \"7092ff60-28f3-4737-897c-41020bc4a5a7\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") "
Oct 31 03:56:37.963624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963572    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-workload-socket\") pod \"0dce71a3-b4c7-459d-92e8-c0515341abf1\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") "
Oct 31 03:56:37.963624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963616    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7092ff60-28f3-4737-897c-41020bc4a5a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7092ff60-28f3-4737-897c-41020bc4a5a7-istiod-ca-cert\") pod \"7092ff60-28f3-4737-897c-41020bc4a5a7\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") "
Oct 31 03:56:37.964028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963632    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-workload-socket" pod "7092ff60-28f3-4737-897c-41020bc4a5a7" (UID: "7092ff60-28f3-4737-897c-41020bc4a5a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.964028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963655    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-credential-socket\") pod \"0dce71a3-b4c7-459d-92e8-c0515341abf1\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") "
Oct 31 03:56:37.964028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963709    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-data" pod "0dce71a3-b4c7-459d-92e8-c0515341abf1" (UID: "0dce71a3-b4c7-459d-92e8-c0515341abf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.964028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963720    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-data" pod "7092ff60-28f3-4737-897c-41020bc4a5a7" (UID: "7092ff60-28f3-4737-897c-41020bc4a5a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.964028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963729    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-token\" (UniqueName: \"kubernetes.io/projected/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-token\") pod \"7092ff60-28f3-4737-897c-41020bc4a5a7\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") "
Oct 31 03:56:37.964028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963746    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dce71a3-b4c7-459d-92e8-c0515341abf1-istiod-ca-cert" pod "0dce71a3-b4c7-459d-92e8-c0515341abf1" (UID: "0dce71a3-b4c7-459d-92e8-c0515341abf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:37.964028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963864    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-workload-certs\") pod \"0dce71a3-b4c7-459d-92e8-c0515341abf1\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") "
Oct 31 03:56:37.964028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963894    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-credential-socket" pod "7092ff60-28f3-4737-897c-41020bc4a5a7" (UID: "7092ff60-28f3-4737-897c-41020bc4a5a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.964028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963914    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-workload-certs\") pod \"7092ff60-28f3-4737-897c-41020bc4a5a7\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") "
Oct 31 03:56:37.964028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963946    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7092ff60-28f3-4737-897c-41020bc4a5a7-kube-api-access-p6dvb\" (UniqueName: \"kubernetes.io/projected/7092ff60-28f3-4737-897c-41020bc4a5a7-kube-api-access-p6dvb\") pod \"7092ff60-28f3-4737-897c-41020bc4a5a7\" (UID: \"7092ff60-28f3-4737-897c-41020bc4a5a7\") "
Oct 31 03:56:37.964028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963976    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-credential-socket" pod "0dce71a3-b4c7-459d-92e8-c0515341abf1" (UID: "0dce71a3-b4c7-459d-92e8-c0515341abf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.964028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.963983    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dce71a3-b4c7-459d-92e8-c0515341abf1-kube-api-access-7m46b\" (UniqueName: \"kubernetes.io/projected/0dce71a3-b4c7-459d-92e8-c0515341abf1-kube-api-access-7m46b\") pod \"0dce71a3-b4c7-459d-92e8-c0515341abf1\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") "
Oct 31 03:56:37.964676 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.964056    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-workload-socket" pod "0dce71a3-b4c7-459d-92e8-c0515341abf1" (UID: "0dce71a3-b4c7-459d-92e8-c0515341abf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.964676 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.964093    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7092ff60-28f3-4737-897c-41020bc4a5a7-istiod-ca-cert" pod "7092ff60-28f3-4737-897c-41020bc4a5a7" (UID: "7092ff60-28f3-4737-897c-41020bc4a5a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:37.964676 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.964099    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-podinfo\") pod \"0dce71a3-b4c7-459d-92e8-c0515341abf1\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") "
Oct 31 03:56:37.964676 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.964280    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-workload-certs" pod "7092ff60-28f3-4737-897c-41020bc4a5a7" (UID: "7092ff60-28f3-4737-897c-41020bc4a5a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.964676 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.964322    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-envoy\") pod \"0dce71a3-b4c7-459d-92e8-c0515341abf1\" (UID: \"0dce71a3-b4c7-459d-92e8-c0515341abf1\") "
Oct 31 03:56:37.964676 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.964473    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-workload-certs" pod "0dce71a3-b4c7-459d-92e8-c0515341abf1" (UID: "0dce71a3-b4c7-459d-92e8-c0515341abf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.965052 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.964992    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm8rl\" (UniqueName: \"kubernetes.io/projected/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-kube-api-access-pm8rl\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965052 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965016    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965052 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965031    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965052 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965047    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965060    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dce71a3-b4c7-459d-92e8-c0515341abf1-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965074    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965089    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965102    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965115    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965144    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965159    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965173    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7092ff60-28f3-4737-897c-41020bc4a5a7-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965187    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965199    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965214    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965229    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.965267 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965243    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a6cc6d-73ae-42a5-899e-c8c05bf010f3-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:37.966022 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.965967    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-podinfo" pod "7092ff60-28f3-4737-897c-41020bc4a5a7" (UID: "7092ff60-28f3-4737-897c-41020bc4a5a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:37.966486 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.966460    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-envoy" pod "7092ff60-28f3-4737-897c-41020bc4a5a7" (UID: "7092ff60-28f3-4737-897c-41020bc4a5a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:37.966605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.966580    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-token" pod "0dce71a3-b4c7-459d-92e8-c0515341abf1" (UID: "0dce71a3-b4c7-459d-92e8-c0515341abf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:37.966605 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.966582    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-podinfo" pod "0dce71a3-b4c7-459d-92e8-c0515341abf1" (UID: "0dce71a3-b4c7-459d-92e8-c0515341abf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:37.966860 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.966843    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-token" pod "7092ff60-28f3-4737-897c-41020bc4a5a7" (UID: "7092ff60-28f3-4737-897c-41020bc4a5a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:37.967010 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.966991    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dce71a3-b4c7-459d-92e8-c0515341abf1-kube-api-access-7m46b" pod "0dce71a3-b4c7-459d-92e8-c0515341abf1" (UID: "0dce71a3-b4c7-459d-92e8-c0515341abf1"). InnerVolumeSpecName "kube-api-access-7m46b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:37.967652 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.967631    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7092ff60-28f3-4737-897c-41020bc4a5a7-kube-api-access-p6dvb" pod "7092ff60-28f3-4737-897c-41020bc4a5a7" (UID: "7092ff60-28f3-4737-897c-41020bc4a5a7"). InnerVolumeSpecName "kube-api-access-p6dvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:37.967965 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:37.967951    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-envoy" pod "0dce71a3-b4c7-459d-92e8-c0515341abf1" (UID: "0dce71a3-b4c7-459d-92e8-c0515341abf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:38.066754 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.066711    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:38.066754 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.066745    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:38.066754 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.066759    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:38.067316 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.066771    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7092ff60-28f3-4737-897c-41020bc4a5a7-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:38.067316 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.066798    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6dvb\" (UniqueName: \"kubernetes.io/projected/7092ff60-28f3-4737-897c-41020bc4a5a7-kube-api-access-p6dvb\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:38.067316 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.066814    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m46b\" (UniqueName: \"kubernetes.io/projected/0dce71a3-b4c7-459d-92e8-c0515341abf1-kube-api-access-7m46b\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:38.067316 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.066826    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:38.067316 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.066838    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dce71a3-b4c7-459d-92e8-c0515341abf1-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:38.091355 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.091316    4900 generic.go:356] "Generic (PLEG): container finished" podID="0dce71a3-b4c7-459d-92e8-c0515341abf1" containerID="ddf4554de2aadfe45045269273bff65d8ce203aac0bf23a945ca3f8932676eda" exitCode=0
Oct 31 03:56:38.091583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.091458    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"
Oct 31 03:56:38.091583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.091469    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx" event={"ID":"0dce71a3-b4c7-459d-92e8-c0515341abf1","Type":"ContainerDied","Data":"ddf4554de2aadfe45045269273bff65d8ce203aac0bf23a945ca3f8932676eda"}
Oct 31 03:56:38.091583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.091523    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx" event={"ID":"0dce71a3-b4c7-459d-92e8-c0515341abf1","Type":"ContainerDied","Data":"a61565008bb1996ccb098faad7f38695b8e8805dcf623c4e6fa94b11911bd1f3"}
Oct 31 03:56:38.091583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.091552    4900 scope.go:122] "RemoveContainer" containerID="ddf4554de2aadfe45045269273bff65d8ce203aac0bf23a945ca3f8932676eda"
Oct 31 03:56:38.095598 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.095392    4900 generic.go:356] "Generic (PLEG): container finished" podID="b38b8385-a404-433d-bcee-5e5ffb33d5da" containerID="86761543486b7d4644fe8db9d0004575d2574f60a12446b38f9e990eca30e2e8" exitCode=0
Oct 31 03:56:38.095598 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.095474    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"
Oct 31 03:56:38.095598 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.095493    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn" event={"ID":"b38b8385-a404-433d-bcee-5e5ffb33d5da","Type":"ContainerDied","Data":"86761543486b7d4644fe8db9d0004575d2574f60a12446b38f9e990eca30e2e8"}
Oct 31 03:56:38.095850 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.095600    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn" event={"ID":"b38b8385-a404-433d-bcee-5e5ffb33d5da","Type":"ContainerDied","Data":"e377a5d2e1f7f5beaf9af5ff4ff2ce49820cdb31ff546412417fca4caa4c11f8"}
Oct 31 03:56:38.123537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.101539    4900 generic.go:356] "Generic (PLEG): container finished" podID="44a6cc6d-73ae-42a5-899e-c8c05bf010f3" containerID="29918f8d23744016edd5b764b1b41f205240fd386218fa2686a0db6c182feda1" exitCode=0
Oct 31 03:56:38.123537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.101646    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd" event={"ID":"44a6cc6d-73ae-42a5-899e-c8c05bf010f3","Type":"ContainerDied","Data":"29918f8d23744016edd5b764b1b41f205240fd386218fa2686a0db6c182feda1"}
Oct 31 03:56:38.123537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.101690    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd" event={"ID":"44a6cc6d-73ae-42a5-899e-c8c05bf010f3","Type":"ContainerDied","Data":"8b3447c8586402aa07b571435fe6e8bc37cc734bbafe07bd2dd53f2e7c0a5aa4"}
Oct 31 03:56:38.123537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.101771    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"
Oct 31 03:56:38.123537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.113070    4900 scope.go:122] "RemoveContainer" containerID="ddf4554de2aadfe45045269273bff65d8ce203aac0bf23a945ca3f8932676eda"
Oct 31 03:56:38.123537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.114711    4900 generic.go:356] "Generic (PLEG): container finished" podID="7092ff60-28f3-4737-897c-41020bc4a5a7" containerID="d4e19d0f8df7d8859832cbc3feaa55fc301c098976db8f75e663ca003b68e24a" exitCode=0
Oct 31 03:56:38.123537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.114775    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"
Oct 31 03:56:38.123537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.114817    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl" event={"ID":"7092ff60-28f3-4737-897c-41020bc4a5a7","Type":"ContainerDied","Data":"d4e19d0f8df7d8859832cbc3feaa55fc301c098976db8f75e663ca003b68e24a"}
Oct 31 03:56:38.123537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.114868    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl" event={"ID":"7092ff60-28f3-4737-897c-41020bc4a5a7","Type":"ContainerDied","Data":"32d503ade1f24838a60817626fd89777fa14cf0881354258cac2fb14d6dc9b3b"}
Oct 31 03:56:38.123537 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:56:38.115647    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddf4554de2aadfe45045269273bff65d8ce203aac0bf23a945ca3f8932676eda\": container with ID starting with ddf4554de2aadfe45045269273bff65d8ce203aac0bf23a945ca3f8932676eda not found: ID does not exist" containerID="ddf4554de2aadfe45045269273bff65d8ce203aac0bf23a945ca3f8932676eda"
Oct 31 03:56:38.123537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.115683    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddf4554de2aadfe45045269273bff65d8ce203aac0bf23a945ca3f8932676eda"} err="failed to get container status \"ddf4554de2aadfe45045269273bff65d8ce203aac0bf23a945ca3f8932676eda\": rpc error: code = NotFound desc = could not find container \"ddf4554de2aadfe45045269273bff65d8ce203aac0bf23a945ca3f8932676eda\": container with ID starting with ddf4554de2aadfe45045269273bff65d8ce203aac0bf23a945ca3f8932676eda not found: ID does not exist"
Oct 31 03:56:38.123537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.115709    4900 scope.go:122] "RemoveContainer" containerID="86761543486b7d4644fe8db9d0004575d2574f60a12446b38f9e990eca30e2e8"
Oct 31 03:56:38.127471 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.127450    4900 scope.go:122] "RemoveContainer" containerID="86761543486b7d4644fe8db9d0004575d2574f60a12446b38f9e990eca30e2e8"
Oct 31 03:56:38.127824 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:56:38.127797    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86761543486b7d4644fe8db9d0004575d2574f60a12446b38f9e990eca30e2e8\": container with ID starting with 86761543486b7d4644fe8db9d0004575d2574f60a12446b38f9e990eca30e2e8 not found: ID does not exist" containerID="86761543486b7d4644fe8db9d0004575d2574f60a12446b38f9e990eca30e2e8"
Oct 31 03:56:38.127900 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.127831    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86761543486b7d4644fe8db9d0004575d2574f60a12446b38f9e990eca30e2e8"} err="failed to get container status \"86761543486b7d4644fe8db9d0004575d2574f60a12446b38f9e990eca30e2e8\": rpc error: code = NotFound desc = could not find container \"86761543486b7d4644fe8db9d0004575d2574f60a12446b38f9e990eca30e2e8\": container with ID starting with 86761543486b7d4644fe8db9d0004575d2574f60a12446b38f9e990eca30e2e8 not found: ID does not exist"
Oct 31 03:56:38.127900 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.127851    4900 scope.go:122] "RemoveContainer" containerID="29918f8d23744016edd5b764b1b41f205240fd386218fa2686a0db6c182feda1"
Oct 31 03:56:38.137613 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.137589    4900 scope.go:122] "RemoveContainer" containerID="29918f8d23744016edd5b764b1b41f205240fd386218fa2686a0db6c182feda1"
Oct 31 03:56:38.137968 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:56:38.137947    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29918f8d23744016edd5b764b1b41f205240fd386218fa2686a0db6c182feda1\": container with ID starting with 29918f8d23744016edd5b764b1b41f205240fd386218fa2686a0db6c182feda1 not found: ID does not exist" containerID="29918f8d23744016edd5b764b1b41f205240fd386218fa2686a0db6c182feda1"
Oct 31 03:56:38.138050 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.137976    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29918f8d23744016edd5b764b1b41f205240fd386218fa2686a0db6c182feda1"} err="failed to get container status \"29918f8d23744016edd5b764b1b41f205240fd386218fa2686a0db6c182feda1\": rpc error: code = NotFound desc = could not find container \"29918f8d23744016edd5b764b1b41f205240fd386218fa2686a0db6c182feda1\": container with ID starting with 29918f8d23744016edd5b764b1b41f205240fd386218fa2686a0db6c182feda1 not found: ID does not exist"
Oct 31 03:56:38.138050 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.137998    4900 scope.go:122] "RemoveContainer" containerID="d4e19d0f8df7d8859832cbc3feaa55fc301c098976db8f75e663ca003b68e24a"
Oct 31 03:56:38.147987 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.147965    4900 scope.go:122] "RemoveContainer" containerID="d4e19d0f8df7d8859832cbc3feaa55fc301c098976db8f75e663ca003b68e24a"
Oct 31 03:56:38.148315 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:56:38.148286    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e19d0f8df7d8859832cbc3feaa55fc301c098976db8f75e663ca003b68e24a\": container with ID starting with d4e19d0f8df7d8859832cbc3feaa55fc301c098976db8f75e663ca003b68e24a not found: ID does not exist" containerID="d4e19d0f8df7d8859832cbc3feaa55fc301c098976db8f75e663ca003b68e24a"
Oct 31 03:56:38.148372 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.148324    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e19d0f8df7d8859832cbc3feaa55fc301c098976db8f75e663ca003b68e24a"} err="failed to get container status \"d4e19d0f8df7d8859832cbc3feaa55fc301c098976db8f75e663ca003b68e24a\": rpc error: code = NotFound desc = could not find container \"d4e19d0f8df7d8859832cbc3feaa55fc301c098976db8f75e663ca003b68e24a\": container with ID starting with d4e19d0f8df7d8859832cbc3feaa55fc301c098976db8f75e663ca003b68e24a not found: ID does not exist"
Oct 31 03:56:38.326355 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.326320    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"]
Oct 31 03:56:38.331593 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.331565    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-4gbwl"]
Oct 31 03:56:38.440366 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.440337    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"]
Oct 31 03:56:38.443175 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.443146    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-qmspd"]
Oct 31 03:56:38.553046 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.553007    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"]
Oct 31 03:56:38.559264 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.559237    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-md2rx"]
Oct 31 03:56:38.667865 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.667753    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"]
Oct 31 03:56:38.671422 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:38.671392    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-97vbn"]
Oct 31 03:56:39.152514 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:39.152475    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dce71a3-b4c7-459d-92e8-c0515341abf1" path="/var/lib/kubelet/pods/0dce71a3-b4c7-459d-92e8-c0515341abf1/volumes"
Oct 31 03:56:39.153066 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:39.153050    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a6cc6d-73ae-42a5-899e-c8c05bf010f3" path="/var/lib/kubelet/pods/44a6cc6d-73ae-42a5-899e-c8c05bf010f3/volumes"
Oct 31 03:56:39.153532 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:39.153518    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7092ff60-28f3-4737-897c-41020bc4a5a7" path="/var/lib/kubelet/pods/7092ff60-28f3-4737-897c-41020bc4a5a7/volumes"
Oct 31 03:56:39.154432 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:39.154416    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b38b8385-a404-433d-bcee-5e5ffb33d5da" path="/var/lib/kubelet/pods/b38b8385-a404-433d-bcee-5e5ffb33d5da/volumes"
Oct 31 03:56:40.694549 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:40.694278    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:42.719977 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:42.719937    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:47.592121 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:47.592085    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:52.481045 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:52.480956    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:57.601009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:57.600959    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:56:58.885920 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:56:58.885886    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:02.427601 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:02.427570    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:06.984577 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:57:06.984531    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:57:07.397065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:07.396969    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:10.669233 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:10.669148    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:11.829882 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:11.829837    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:12.588493 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:12.588455    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:17.475880 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:17.475842    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:17.698670 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:17.698613    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:22.345483 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:22.345449    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:27.190003 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:27.189967    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:28.729656 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:28.729579    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:57:28.741746 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:28.741721    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 03:57:32.190223 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.190186    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"]
Oct 31 03:57:32.190933 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.190913    4900 cpu_manager.go:414] "containerMap: removing container" podUID="7092ff60-28f3-4737-897c-41020bc4a5a7" containerName="istio-proxy"
Oct 31 03:57:32.191019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.190934    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7092ff60-28f3-4737-897c-41020bc4a5a7" containerName="istio-proxy"
Oct 31 03:57:32.191019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.190958    4900 cpu_manager.go:414] "containerMap: removing container" podUID="b38b8385-a404-433d-bcee-5e5ffb33d5da" containerName="istio-proxy"
Oct 31 03:57:32.191019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.190963    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38b8385-a404-433d-bcee-5e5ffb33d5da" containerName="istio-proxy"
Oct 31 03:57:32.191019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.190984    4900 cpu_manager.go:414] "containerMap: removing container" podUID="44a6cc6d-73ae-42a5-899e-c8c05bf010f3" containerName="istio-proxy"
Oct 31 03:57:32.191019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.190989    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a6cc6d-73ae-42a5-899e-c8c05bf010f3" containerName="istio-proxy"
Oct 31 03:57:32.191019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.191002    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0dce71a3-b4c7-459d-92e8-c0515341abf1" containerName="istio-proxy"
Oct 31 03:57:32.191019 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.191007    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dce71a3-b4c7-459d-92e8-c0515341abf1" containerName="istio-proxy"
Oct 31 03:57:32.191273 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.191130    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38b8385-a404-433d-bcee-5e5ffb33d5da" containerName="istio-proxy"
Oct 31 03:57:32.191273 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.191149    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a6cc6d-73ae-42a5-899e-c8c05bf010f3" containerName="istio-proxy"
Oct 31 03:57:32.191273 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.191162    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dce71a3-b4c7-459d-92e8-c0515341abf1" containerName="istio-proxy"
Oct 31 03:57:32.191273 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.191172    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="7092ff60-28f3-4737-897c-41020bc4a5a7" containerName="istio-proxy"
Oct 31 03:57:32.195910 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.195889    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.242104 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.242072    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:32.247035 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.247009    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"]
Oct 31 03:57:32.294189 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.294163    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c0d50c\"/\"openshift-service-ca.crt\""
Oct 31 03:57:32.294189 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.294173    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-c0d50c\"/\"kube-root-ca.crt\""
Oct 31 03:57:32.294366 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.294174    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c0d50c\"/\"default-dockercfg-z785j\""
Oct 31 03:57:32.297984 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.297966    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-c0d50c\"/\"auth-test-model-c0d50c-kserve-self-signed-certs\""
Oct 31 03:57:32.312963 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.312931    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-home\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.313101 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.313059    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-tmp-dir\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.313167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.313118    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-dshm\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.313167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.313144    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swnrg\" (UniqueName: \"kubernetes.io/projected/d682228b-0707-4cf7-9783-79726934b6f9-kube-api-access-swnrg\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.313277 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.313206    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-model-cache\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.313337 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.313279    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d682228b-0707-4cf7-9783-79726934b6f9-tls-certs\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.414885 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.414853    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-home\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.415070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.414952    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-tmp-dir\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.415070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.414988    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-dshm\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.415070 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.415015    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swnrg\" (UniqueName: \"kubernetes.io/projected/d682228b-0707-4cf7-9783-79726934b6f9-kube-api-access-swnrg\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.415237 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.415142    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-model-cache\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.415295 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.415245    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d682228b-0707-4cf7-9783-79726934b6f9-tls-certs\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.415346 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.415315    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-home\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.415346 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.415333    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-tmp-dir\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.415656 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.415628    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-model-cache\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.417295 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.417271    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-dshm\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.417555 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.417539    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d682228b-0707-4cf7-9783-79726934b6f9-tls-certs\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.460657 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.460593    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:32.472156 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.472134    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swnrg\" (UniqueName: \"kubernetes.io/projected/d682228b-0707-4cf7-9783-79726934b6f9-kube-api-access-swnrg\") pod \"auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") " pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.508167 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.508139    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:32.877319 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:57:32.877284    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd682228b_0707_4cf7_9783_79726934b6f9.slice/crio-cef28a435d93afb7994b85bb3fc466211e265fe655a3b8be36d6434352c5a582 WatchSource:0}: Error finding container cef28a435d93afb7994b85bb3fc466211e265fe655a3b8be36d6434352c5a582: Status 404 returned error can't find the container with id cef28a435d93afb7994b85bb3fc466211e265fe655a3b8be36d6434352c5a582
Oct 31 03:57:32.879965 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:32.879864    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"]
Oct 31 03:57:33.443004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:33.442951    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj" event={"ID":"d682228b-0707-4cf7-9783-79726934b6f9","Type":"ContainerStarted","Data":"2b3c587b3f5f2be7d3cf3319d907133bdce5407650f1126bce8ed4b6f36d691d"}
Oct 31 03:57:33.443004 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:33.443006    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj" event={"ID":"d682228b-0707-4cf7-9783-79726934b6f9","Type":"ContainerStarted","Data":"cef28a435d93afb7994b85bb3fc466211e265fe655a3b8be36d6434352c5a582"}
Oct 31 03:57:33.508500 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:33.508452    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:33.560590 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:33.560538    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj" podStartSLOduration=1.5605232980000001 podStartE2EDuration="1.560523298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:57:33.559367679 +0000 UTC m=+3006.525701024" watchObservedRunningTime="2026-10-31 03:57:33.560523298 +0000 UTC m=+3006.526856641"
Oct 31 03:57:36.984872 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:57:36.984833    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:57:37.194164 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:37.194136    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:42.090028 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:42.089992    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:42.508865 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:42.508828    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:42.521382 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:42.521355    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:43.514985 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:43.514954    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 03:57:46.971834 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:46.971771    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:51.797172 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:51.797141    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:56.691478 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:56.691436    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:59.283265 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:59.283226    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:57:59.517555 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:57:59.517523    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:00.608391 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:00.608348    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:01.787110 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:01.787072    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:05.484272 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:05.484232    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:06.631863 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:06.631830    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:06.998221 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:58:06.998172    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:58:08.032474 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:08.032437    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"]
Oct 31 03:58:08.033017 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:08.032649    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p" podUID="eaea4507-660f-4b97-b8a7-d349c4441251" containerName="istio-proxy" containerID="cri-o://476e26dc39e6c9f18225dd6f0bc00c7ea8a92f14d9fb55a433585ffbb1e96912" gracePeriod=30
Oct 31 03:58:11.576037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:11.576006    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:13.308578 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.308523    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:58:13.429083 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.429054    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-istio-envoy\") pod \"eaea4507-660f-4b97-b8a7-d349c4441251\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") "
Oct 31 03:58:13.429232 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.429115    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-istio-data\") pod \"eaea4507-660f-4b97-b8a7-d349c4441251\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") "
Oct 31 03:58:13.429232 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.429161    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-credential-socket\") pod \"eaea4507-660f-4b97-b8a7-d349c4441251\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") "
Oct 31 03:58:13.429232 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.429217    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaea4507-660f-4b97-b8a7-d349c4441251-istio-token\" (UniqueName: \"kubernetes.io/projected/eaea4507-660f-4b97-b8a7-d349c4441251-istio-token\") pod \"eaea4507-660f-4b97-b8a7-d349c4441251\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") "
Oct 31 03:58:13.429361 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.429242    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-workload-socket\") pod \"eaea4507-660f-4b97-b8a7-d349c4441251\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") "
Oct 31 03:58:13.429402 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.429382    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-workload-certs\") pod \"eaea4507-660f-4b97-b8a7-d349c4441251\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") "
Oct 31 03:58:13.429402 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.429385    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-istio-data" pod "eaea4507-660f-4b97-b8a7-d349c4441251" (UID: "eaea4507-660f-4b97-b8a7-d349c4441251"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:13.429504 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.429436    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaea4507-660f-4b97-b8a7-d349c4441251-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaea4507-660f-4b97-b8a7-d349c4441251-istio-podinfo\") pod \"eaea4507-660f-4b97-b8a7-d349c4441251\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") "
Oct 31 03:58:13.429504 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.429494    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaea4507-660f-4b97-b8a7-d349c4441251-kube-api-access-v92tl\" (UniqueName: \"kubernetes.io/projected/eaea4507-660f-4b97-b8a7-d349c4441251-kube-api-access-v92tl\") pod \"eaea4507-660f-4b97-b8a7-d349c4441251\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") "
Oct 31 03:58:13.429612 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.429511    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-credential-socket" pod "eaea4507-660f-4b97-b8a7-d349c4441251" (UID: "eaea4507-660f-4b97-b8a7-d349c4441251"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:13.429612 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.429535    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaea4507-660f-4b97-b8a7-d349c4441251-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaea4507-660f-4b97-b8a7-d349c4441251-istiod-ca-cert\") pod \"eaea4507-660f-4b97-b8a7-d349c4441251\" (UID: \"eaea4507-660f-4b97-b8a7-d349c4441251\") "
Oct 31 03:58:13.429612 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.429601    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-workload-certs" pod "eaea4507-660f-4b97-b8a7-d349c4441251" (UID: "eaea4507-660f-4b97-b8a7-d349c4441251"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:13.429868 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.429843    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaea4507-660f-4b97-b8a7-d349c4441251-istiod-ca-cert" pod "eaea4507-660f-4b97-b8a7-d349c4441251" (UID: "eaea4507-660f-4b97-b8a7-d349c4441251"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:58:13.429962 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.429899    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-workload-socket" pod "eaea4507-660f-4b97-b8a7-d349c4441251" (UID: "eaea4507-660f-4b97-b8a7-d349c4441251"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:13.430150 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.430126    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:13.430235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.430156    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaea4507-660f-4b97-b8a7-d349c4441251-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:13.430235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.430171    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:13.430235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.430187    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:13.430235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.430203    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:13.431468 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.431445    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaea4507-660f-4b97-b8a7-d349c4441251-istio-token" pod "eaea4507-660f-4b97-b8a7-d349c4441251" (UID: "eaea4507-660f-4b97-b8a7-d349c4441251"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:13.431555 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.431468    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-istio-envoy" pod "eaea4507-660f-4b97-b8a7-d349c4441251" (UID: "eaea4507-660f-4b97-b8a7-d349c4441251"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:13.431603 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.431584    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaea4507-660f-4b97-b8a7-d349c4441251-kube-api-access-v92tl" pod "eaea4507-660f-4b97-b8a7-d349c4441251" (UID: "eaea4507-660f-4b97-b8a7-d349c4441251"). InnerVolumeSpecName "kube-api-access-v92tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:13.431767 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.431749    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaea4507-660f-4b97-b8a7-d349c4441251-istio-podinfo" pod "eaea4507-660f-4b97-b8a7-d349c4441251" (UID: "eaea4507-660f-4b97-b8a7-d349c4441251"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:58:13.531656 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.531626    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaea4507-660f-4b97-b8a7-d349c4441251-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:13.531656 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.531652    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v92tl\" (UniqueName: \"kubernetes.io/projected/eaea4507-660f-4b97-b8a7-d349c4441251-kube-api-access-v92tl\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:13.531656 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.531663    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaea4507-660f-4b97-b8a7-d349c4441251-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:13.531918 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.531672    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaea4507-660f-4b97-b8a7-d349c4441251-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:13.690067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.689983    4900 generic.go:356] "Generic (PLEG): container finished" podID="eaea4507-660f-4b97-b8a7-d349c4441251" containerID="476e26dc39e6c9f18225dd6f0bc00c7ea8a92f14d9fb55a433585ffbb1e96912" exitCode=0
Oct 31 03:58:13.690067 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.690024    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"
Oct 31 03:58:13.690294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.690061    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p" event={"ID":"eaea4507-660f-4b97-b8a7-d349c4441251","Type":"ContainerDied","Data":"476e26dc39e6c9f18225dd6f0bc00c7ea8a92f14d9fb55a433585ffbb1e96912"}
Oct 31 03:58:13.690294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.690113    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p" event={"ID":"eaea4507-660f-4b97-b8a7-d349c4441251","Type":"ContainerDied","Data":"600ee7c0a4b781c8558f52a3407379d0b29e718a701d7f477922e9cd7915bebe"}
Oct 31 03:58:13.690294 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.690134    4900 scope.go:122] "RemoveContainer" containerID="476e26dc39e6c9f18225dd6f0bc00c7ea8a92f14d9fb55a433585ffbb1e96912"
Oct 31 03:58:13.701064 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.701043    4900 scope.go:122] "RemoveContainer" containerID="476e26dc39e6c9f18225dd6f0bc00c7ea8a92f14d9fb55a433585ffbb1e96912"
Oct 31 03:58:13.701304 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:58:13.701283    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476e26dc39e6c9f18225dd6f0bc00c7ea8a92f14d9fb55a433585ffbb1e96912\": container with ID starting with 476e26dc39e6c9f18225dd6f0bc00c7ea8a92f14d9fb55a433585ffbb1e96912 not found: ID does not exist" containerID="476e26dc39e6c9f18225dd6f0bc00c7ea8a92f14d9fb55a433585ffbb1e96912"
Oct 31 03:58:13.701375 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.701315    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476e26dc39e6c9f18225dd6f0bc00c7ea8a92f14d9fb55a433585ffbb1e96912"} err="failed to get container status \"476e26dc39e6c9f18225dd6f0bc00c7ea8a92f14d9fb55a433585ffbb1e96912\": rpc error: code = NotFound desc = could not find container \"476e26dc39e6c9f18225dd6f0bc00c7ea8a92f14d9fb55a433585ffbb1e96912\": container with ID starting with 476e26dc39e6c9f18225dd6f0bc00c7ea8a92f14d9fb55a433585ffbb1e96912 not found: ID does not exist"
Oct 31 03:58:13.855192 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.855161    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"]
Oct 31 03:58:13.858864 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:13.858844    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-pzd8p"]
Oct 31 03:58:14.838700 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:14.838661    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"]
Oct 31 03:58:14.839211 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:14.838911    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb" podUID="cf89487b-5877-4f0e-81f7-abb1ff44d51a" containerName="istio-proxy" containerID="cri-o://36b6df5dcd052d9e9b6f4af182cce74e3addf572b80b9de8d437a7e9e3f14572" gracePeriod=30
Oct 31 03:58:15.153238 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:15.153164    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaea4507-660f-4b97-b8a7-d349c4441251" path="/var/lib/kubelet/pods/eaea4507-660f-4b97-b8a7-d349c4441251/volumes"
Oct 31 03:58:16.453802 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:16.453762    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:19.308615 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:19.308575    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:58:19.309151 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:19.308644    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb" podUID="cf89487b-5877-4f0e-81f7-abb1ff44d51a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:58:20.112056 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.111985    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:58:20.207177 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.207139    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-envoy\") pod \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") "
Oct 31 03:58:20.207377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.207242    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-workload-socket\") pod \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") "
Oct 31 03:58:20.207377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.207310    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-credential-socket\") pod \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") "
Oct 31 03:58:20.207377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.207341    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-workload-certs\") pod \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") "
Oct 31 03:58:20.207377 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.207358    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf89487b-5877-4f0e-81f7-abb1ff44d51a-kube-api-access-7s87r\" (UniqueName: \"kubernetes.io/projected/cf89487b-5877-4f0e-81f7-abb1ff44d51a-kube-api-access-7s87r\") pod \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") "
Oct 31 03:58:20.207613 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.207397    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-podinfo\") pod \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") "
Oct 31 03:58:20.207613 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.207433    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-data\") pod \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") "
Oct 31 03:58:20.207613 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.207463    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-token\" (UniqueName: \"kubernetes.io/projected/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-token\") pod \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") "
Oct 31 03:58:20.207613 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.207503    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istiod-ca-cert\") pod \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\" (UID: \"cf89487b-5877-4f0e-81f7-abb1ff44d51a\") "
Oct 31 03:58:20.207613 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.207568    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-workload-socket" pod "cf89487b-5877-4f0e-81f7-abb1ff44d51a" (UID: "cf89487b-5877-4f0e-81f7-abb1ff44d51a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:20.207916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.207757    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-data" pod "cf89487b-5877-4f0e-81f7-abb1ff44d51a" (UID: "cf89487b-5877-4f0e-81f7-abb1ff44d51a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:20.207916 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.207766    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-workload-certs" pod "cf89487b-5877-4f0e-81f7-abb1ff44d51a" (UID: "cf89487b-5877-4f0e-81f7-abb1ff44d51a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:20.208026 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.207933    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-credential-socket" pod "cf89487b-5877-4f0e-81f7-abb1ff44d51a" (UID: "cf89487b-5877-4f0e-81f7-abb1ff44d51a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:20.208355 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.208210    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istiod-ca-cert" pod "cf89487b-5877-4f0e-81f7-abb1ff44d51a" (UID: "cf89487b-5877-4f0e-81f7-abb1ff44d51a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:58:20.208844 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.208819    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.208844 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.208843    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.209060 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.208852    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.209060 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.208864    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.209060 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.208877    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.209967 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.209930    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-envoy" pod "cf89487b-5877-4f0e-81f7-abb1ff44d51a" (UID: "cf89487b-5877-4f0e-81f7-abb1ff44d51a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:58:20.210215 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.210189    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-podinfo" pod "cf89487b-5877-4f0e-81f7-abb1ff44d51a" (UID: "cf89487b-5877-4f0e-81f7-abb1ff44d51a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:58:20.210330 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.210314    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-token" pod "cf89487b-5877-4f0e-81f7-abb1ff44d51a" (UID: "cf89487b-5877-4f0e-81f7-abb1ff44d51a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:20.210401 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.210356    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf89487b-5877-4f0e-81f7-abb1ff44d51a-kube-api-access-7s87r" pod "cf89487b-5877-4f0e-81f7-abb1ff44d51a" (UID: "cf89487b-5877-4f0e-81f7-abb1ff44d51a"). InnerVolumeSpecName "kube-api-access-7s87r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:58:20.310537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.310490    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.310537 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.310536    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.310980 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.310551    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf89487b-5877-4f0e-81f7-abb1ff44d51a-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.310980 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.310565    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s87r\" (UniqueName: \"kubernetes.io/projected/cf89487b-5877-4f0e-81f7-abb1ff44d51a-kube-api-access-7s87r\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:58:20.734179 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.734146    4900 generic.go:356] "Generic (PLEG): container finished" podID="cf89487b-5877-4f0e-81f7-abb1ff44d51a" containerID="36b6df5dcd052d9e9b6f4af182cce74e3addf572b80b9de8d437a7e9e3f14572" exitCode=0
Oct 31 03:58:20.734405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.734194    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"
Oct 31 03:58:20.734405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.734233    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb" event={"ID":"cf89487b-5877-4f0e-81f7-abb1ff44d51a","Type":"ContainerDied","Data":"36b6df5dcd052d9e9b6f4af182cce74e3addf572b80b9de8d437a7e9e3f14572"}
Oct 31 03:58:20.734405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.734295    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb" event={"ID":"cf89487b-5877-4f0e-81f7-abb1ff44d51a","Type":"ContainerDied","Data":"82c28a71be47235ac643e8a4c3426b2ee45b11128127aa35ceb7c0607a5dd03b"}
Oct 31 03:58:20.734405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.734325    4900 scope.go:122] "RemoveContainer" containerID="36b6df5dcd052d9e9b6f4af182cce74e3addf572b80b9de8d437a7e9e3f14572"
Oct 31 03:58:20.748507 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.748485    4900 scope.go:122] "RemoveContainer" containerID="36b6df5dcd052d9e9b6f4af182cce74e3addf572b80b9de8d437a7e9e3f14572"
Oct 31 03:58:20.748841 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:58:20.748818    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b6df5dcd052d9e9b6f4af182cce74e3addf572b80b9de8d437a7e9e3f14572\": container with ID starting with 36b6df5dcd052d9e9b6f4af182cce74e3addf572b80b9de8d437a7e9e3f14572 not found: ID does not exist" containerID="36b6df5dcd052d9e9b6f4af182cce74e3addf572b80b9de8d437a7e9e3f14572"
Oct 31 03:58:20.748892 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.748852    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b6df5dcd052d9e9b6f4af182cce74e3addf572b80b9de8d437a7e9e3f14572"} err="failed to get container status \"36b6df5dcd052d9e9b6f4af182cce74e3addf572b80b9de8d437a7e9e3f14572\": rpc error: code = NotFound desc = could not find container \"36b6df5dcd052d9e9b6f4af182cce74e3addf572b80b9de8d437a7e9e3f14572\": container with ID starting with 36b6df5dcd052d9e9b6f4af182cce74e3addf572b80b9de8d437a7e9e3f14572 not found: ID does not exist"
Oct 31 03:58:20.900901 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.900860    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"]
Oct 31 03:58:20.951206 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:20.951168    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-khbhb"]
Oct 31 03:58:21.153635 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:21.153558    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf89487b-5877-4f0e-81f7-abb1ff44d51a" path="/var/lib/kubelet/pods/cf89487b-5877-4f0e-81f7-abb1ff44d51a/volumes"
Oct 31 03:58:21.369409 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:21.369378    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:26.273284 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:26.273245    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:31.126924 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:31.126891    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:36.147304 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:36.147272    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:36.985000 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:58:36.984966    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:58:40.982625 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:40.982551    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:45.893212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:45.893178    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:48.668629 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:48.668597    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:49.129125 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:49.129082    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:50.796009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:50.795972    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:54.146097 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:54.146057    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:55.671133 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:55.671097    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:58:59.620149 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:58:59.620115    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:00.161212 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.161177    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"]
Oct 31 03:59:00.161398 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.161367    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d" podUID="8e1017c9-1f08-4441-adfd-90415b9c6f17" containerName="discovery" containerID="cri-o://98c64a37313e85ccde311649ccce5838ee4c239ddee757ba3d54c005aedfb07f" gracePeriod=30
Oct 31 03:59:00.415743 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.415684    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:59:00.491874 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.491839    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-kubeconfig\") pod \"8e1017c9-1f08-4441-adfd-90415b9c6f17\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") "
Oct 31 03:59:00.492065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.491907    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1017c9-1f08-4441-adfd-90415b9c6f17-kube-api-access-hcfqh\" (UniqueName: \"kubernetes.io/projected/8e1017c9-1f08-4441-adfd-90415b9c6f17-kube-api-access-hcfqh\") pod \"8e1017c9-1f08-4441-adfd-90415b9c6f17\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") "
Oct 31 03:59:00.492065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.491965    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-csr-dns-cert\") pod \"8e1017c9-1f08-4441-adfd-90415b9c6f17\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") "
Oct 31 03:59:00.492065 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.491981    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-cacerts\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-cacerts\") pod \"8e1017c9-1f08-4441-adfd-90415b9c6f17\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") "
Oct 31 03:59:00.492251 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.492113    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-token\" (UniqueName: \"kubernetes.io/projected/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-token\") pod \"8e1017c9-1f08-4441-adfd-90415b9c6f17\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") "
Oct 31 03:59:00.492251 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.492201    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-csr-ca-configmap\") pod \"8e1017c9-1f08-4441-adfd-90415b9c6f17\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") "
Oct 31 03:59:00.492251 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.492241    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e1017c9-1f08-4441-adfd-90415b9c6f17-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1017c9-1f08-4441-adfd-90415b9c6f17-local-certs\") pod \"8e1017c9-1f08-4441-adfd-90415b9c6f17\" (UID: \"8e1017c9-1f08-4441-adfd-90415b9c6f17\") "
Oct 31 03:59:00.492655 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.492617    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-csr-ca-configmap" pod "8e1017c9-1f08-4441-adfd-90415b9c6f17" (UID: "8e1017c9-1f08-4441-adfd-90415b9c6f17"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:00.492859 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.492835    4900 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-csr-ca-configmap\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:00.494198 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.494176    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-kubeconfig" pod "8e1017c9-1f08-4441-adfd-90415b9c6f17" (UID: "8e1017c9-1f08-4441-adfd-90415b9c6f17"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:00.494341 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.494317    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-cacerts" pod "8e1017c9-1f08-4441-adfd-90415b9c6f17" (UID: "8e1017c9-1f08-4441-adfd-90415b9c6f17"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:00.494522 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.494501    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-csr-dns-cert" pod "8e1017c9-1f08-4441-adfd-90415b9c6f17" (UID: "8e1017c9-1f08-4441-adfd-90415b9c6f17"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:00.494801 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.494769    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1017c9-1f08-4441-adfd-90415b9c6f17-kube-api-access-hcfqh" pod "8e1017c9-1f08-4441-adfd-90415b9c6f17" (UID: "8e1017c9-1f08-4441-adfd-90415b9c6f17"). InnerVolumeSpecName "kube-api-access-hcfqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:00.494889 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.494831    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-token" pod "8e1017c9-1f08-4441-adfd-90415b9c6f17" (UID: "8e1017c9-1f08-4441-adfd-90415b9c6f17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:00.494889 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.494832    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e1017c9-1f08-4441-adfd-90415b9c6f17-local-certs" pod "8e1017c9-1f08-4441-adfd-90415b9c6f17" (UID: "8e1017c9-1f08-4441-adfd-90415b9c6f17"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:00.571455 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.571424    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:00.594144 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.594110    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:00.594144 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.594137    4900 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e1017c9-1f08-4441-adfd-90415b9c6f17-local-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:00.594338 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.594150    4900 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-kubeconfig\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:00.594338 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.594165    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcfqh\" (UniqueName: \"kubernetes.io/projected/8e1017c9-1f08-4441-adfd-90415b9c6f17-kube-api-access-hcfqh\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:00.594338 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.594181    4900 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-istio-csr-dns-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:00.594338 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.594192    4900 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/8e1017c9-1f08-4441-adfd-90415b9c6f17-cacerts\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:00.975274 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.975236    4900 generic.go:356] "Generic (PLEG): container finished" podID="8e1017c9-1f08-4441-adfd-90415b9c6f17" containerID="98c64a37313e85ccde311649ccce5838ee4c239ddee757ba3d54c005aedfb07f" exitCode=0
Oct 31 03:59:00.975685 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.975279    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"
Oct 31 03:59:00.975685 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.975279    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d" event={"ID":"8e1017c9-1f08-4441-adfd-90415b9c6f17","Type":"ContainerDied","Data":"98c64a37313e85ccde311649ccce5838ee4c239ddee757ba3d54c005aedfb07f"}
Oct 31 03:59:00.975685 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.975401    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d" event={"ID":"8e1017c9-1f08-4441-adfd-90415b9c6f17","Type":"ContainerDied","Data":"14e9a83d6243af7b83bb82b85d9909dd3c95b74863a7f1ee6b413a2867d4c4ca"}
Oct 31 03:59:00.975685 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.975428    4900 scope.go:122] "RemoveContainer" containerID="98c64a37313e85ccde311649ccce5838ee4c239ddee757ba3d54c005aedfb07f"
Oct 31 03:59:00.985607 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.985589    4900 scope.go:122] "RemoveContainer" containerID="98c64a37313e85ccde311649ccce5838ee4c239ddee757ba3d54c005aedfb07f"
Oct 31 03:59:00.985921 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:59:00.985903    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98c64a37313e85ccde311649ccce5838ee4c239ddee757ba3d54c005aedfb07f\": container with ID starting with 98c64a37313e85ccde311649ccce5838ee4c239ddee757ba3d54c005aedfb07f not found: ID does not exist" containerID="98c64a37313e85ccde311649ccce5838ee4c239ddee757ba3d54c005aedfb07f"
Oct 31 03:59:00.985970 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:00.985929    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98c64a37313e85ccde311649ccce5838ee4c239ddee757ba3d54c005aedfb07f"} err="failed to get container status \"98c64a37313e85ccde311649ccce5838ee4c239ddee757ba3d54c005aedfb07f\": rpc error: code = NotFound desc = could not find container \"98c64a37313e85ccde311649ccce5838ee4c239ddee757ba3d54c005aedfb07f\": container with ID starting with 98c64a37313e85ccde311649ccce5838ee4c239ddee757ba3d54c005aedfb07f not found: ID does not exist"
Oct 31 03:59:01.154260 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:01.154222    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"]
Oct 31 03:59:01.200583 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:01.200543    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-cmg7d"]
Oct 31 03:59:03.152812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:03.152754    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1017c9-1f08-4441-adfd-90415b9c6f17" path="/var/lib/kubelet/pods/8e1017c9-1f08-4441-adfd-90415b9c6f17/volumes"
Oct 31 03:59:03.522187 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:03.522154    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"]
Oct 31 03:59:03.522374 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:03.522356    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm" podUID="7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" containerName="istio-proxy" containerID="cri-o://4721740c281a96ca29d3adb400b4060fc513d1aac334e3fe077b221b4c82eedf" gracePeriod=30
Oct 31 03:59:05.486196 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:05.486155    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:06.984079 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:59:06.984040    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:59:07.500682 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:07.500646    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:07.593889 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:07.593857    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"]
Oct 31 03:59:07.594077 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:07.594042    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" podUID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" containerName="istio-proxy" containerID="cri-o://ea601a4aa2c83cd331c7186d88c654921acdc374ae199454f1114902633dd41b" gracePeriod=30
Oct 31 03:59:08.776634 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.776603    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:59:08.893577 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.893499    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-credential-socket\") pod \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") "
Oct 31 03:59:08.893577 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.893562    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-workload-certs\") pod \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") "
Oct 31 03:59:08.893776 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.893611    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-kube-api-access-8vrlq\" (UniqueName: \"kubernetes.io/projected/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-kube-api-access-8vrlq\") pod \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") "
Oct 31 03:59:08.893776 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.893688    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istiod-ca-cert\") pod \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") "
Oct 31 03:59:08.893776 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.893726    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-workload-socket\") pod \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") "
Oct 31 03:59:08.893776 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.893762    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-envoy\") pod \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") "
Oct 31 03:59:08.894029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.893861    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-podinfo\") pod \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") "
Oct 31 03:59:08.894029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.893889    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-workload-certs" pod "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" (UID: "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:08.894029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.893964    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-data\") pod \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") "
Oct 31 03:59:08.894029 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.894017    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-token\" (UniqueName: \"kubernetes.io/projected/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-token\") pod \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\" (UID: \"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a\") "
Oct 31 03:59:08.894235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.894057    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-workload-socket" pod "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" (UID: "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:08.894235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.894108    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istiod-ca-cert" pod "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" (UID: "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:08.894235 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.893827    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-credential-socket" pod "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" (UID: "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:08.894372 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.894252    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-data" pod "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" (UID: "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:08.894624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.894598    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.894624 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.894626    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.894835 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.894642    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.894835 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.894657    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.894835 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.894671    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.895975 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.895951    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-kube-api-access-8vrlq" pod "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" (UID: "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a"). InnerVolumeSpecName "kube-api-access-8vrlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:08.896091 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.896072    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-envoy" pod "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" (UID: "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:08.896141 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.896083    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-podinfo" pod "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" (UID: "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:59:08.896141 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.896126    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-token" pod "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" (UID: "7d719ccc-8bf1-4e14-a62e-0dcea593ed7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:08.995582 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.995550    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.995582 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.995582    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.995764 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.995592    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.995764 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:08.995602    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vrlq\" (UniqueName: \"kubernetes.io/projected/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a-kube-api-access-8vrlq\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:09.022481 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:09.022447    4900 generic.go:356] "Generic (PLEG): container finished" podID="7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" containerID="4721740c281a96ca29d3adb400b4060fc513d1aac334e3fe077b221b4c82eedf" exitCode=0
Oct 31 03:59:09.022654 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:09.022530    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"
Oct 31 03:59:09.022654 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:09.022525    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm" event={"ID":"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a","Type":"ContainerDied","Data":"4721740c281a96ca29d3adb400b4060fc513d1aac334e3fe077b221b4c82eedf"}
Oct 31 03:59:09.022775 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:09.022659    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm" event={"ID":"7d719ccc-8bf1-4e14-a62e-0dcea593ed7a","Type":"ContainerDied","Data":"81d493542dd79fb844556ddb801e94547acfc4c69c56d540f5ed4733a60ece68"}
Oct 31 03:59:09.022775 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:09.022682    4900 scope.go:122] "RemoveContainer" containerID="4721740c281a96ca29d3adb400b4060fc513d1aac334e3fe077b221b4c82eedf"
Oct 31 03:59:09.032717 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:09.032700    4900 scope.go:122] "RemoveContainer" containerID="4721740c281a96ca29d3adb400b4060fc513d1aac334e3fe077b221b4c82eedf"
Oct 31 03:59:09.033016 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:59:09.032998    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4721740c281a96ca29d3adb400b4060fc513d1aac334e3fe077b221b4c82eedf\": container with ID starting with 4721740c281a96ca29d3adb400b4060fc513d1aac334e3fe077b221b4c82eedf not found: ID does not exist" containerID="4721740c281a96ca29d3adb400b4060fc513d1aac334e3fe077b221b4c82eedf"
Oct 31 03:59:09.033088 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:09.033026    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4721740c281a96ca29d3adb400b4060fc513d1aac334e3fe077b221b4c82eedf"} err="failed to get container status \"4721740c281a96ca29d3adb400b4060fc513d1aac334e3fe077b221b4c82eedf\": rpc error: code = NotFound desc = could not find container \"4721740c281a96ca29d3adb400b4060fc513d1aac334e3fe077b221b4c82eedf\": container with ID starting with 4721740c281a96ca29d3adb400b4060fc513d1aac334e3fe077b221b4c82eedf not found: ID does not exist"
Oct 31 03:59:09.192900 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:09.192814    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"]
Oct 31 03:59:09.195071 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:09.195047    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b59563-openshift-default-6df66b7897-hd5jm"]
Oct 31 03:59:10.493011 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:10.492975    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:11.153798 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:11.153761    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" path="/var/lib/kubelet/pods/7d719ccc-8bf1-4e14-a62e-0dcea593ed7a/volumes"
Oct 31 03:59:12.860828 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.860745    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:59:12.938053 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.938025    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-workload-certs\") pod \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") "
Oct 31 03:59:12.938196 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.938084    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-podinfo\") pod \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") "
Oct 31 03:59:12.938196 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.938104    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-data\") pod \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") "
Oct 31 03:59:12.938196 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.938130    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-envoy\") pod \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") "
Oct 31 03:59:12.938196 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.938187    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-workload-socket\") pod \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") "
Oct 31 03:59:12.938424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.938222    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-credential-socket\") pod \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") "
Oct 31 03:59:12.938424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.938272    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-token\" (UniqueName: \"kubernetes.io/projected/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-token\") pod \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") "
Oct 31 03:59:12.938424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.938278    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-workload-certs" pod "73481f65-a2ac-45c1-be4e-eb29aa0915a9" (UID: "73481f65-a2ac-45c1-be4e-eb29aa0915a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:12.938424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.938372    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73481f65-a2ac-45c1-be4e-eb29aa0915a9-kube-api-access-dhmbs\" (UniqueName: \"kubernetes.io/projected/73481f65-a2ac-45c1-be4e-eb29aa0915a9-kube-api-access-dhmbs\") pod \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") "
Oct 31 03:59:12.938424 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.938416    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-data" pod "73481f65-a2ac-45c1-be4e-eb29aa0915a9" (UID: "73481f65-a2ac-45c1-be4e-eb29aa0915a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:12.938973 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.938455    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istiod-ca-cert\") pod \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\" (UID: \"73481f65-a2ac-45c1-be4e-eb29aa0915a9\") "
Oct 31 03:59:12.938973 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.938491    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-credential-socket" pod "73481f65-a2ac-45c1-be4e-eb29aa0915a9" (UID: "73481f65-a2ac-45c1-be4e-eb29aa0915a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:12.938973 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.938741    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-workload-socket" pod "73481f65-a2ac-45c1-be4e-eb29aa0915a9" (UID: "73481f65-a2ac-45c1-be4e-eb29aa0915a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:12.938973 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.938872    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istiod-ca-cert" pod "73481f65-a2ac-45c1-be4e-eb29aa0915a9" (UID: "73481f65-a2ac-45c1-be4e-eb29aa0915a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:12.939248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.939024    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:12.939248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.939042    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:12.939248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.939052    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:12.939248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.939060    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:12.939248 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.939069    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:12.940364 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.940331    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-podinfo" pod "73481f65-a2ac-45c1-be4e-eb29aa0915a9" (UID: "73481f65-a2ac-45c1-be4e-eb29aa0915a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:59:12.940460 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.940428    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-envoy" pod "73481f65-a2ac-45c1-be4e-eb29aa0915a9" (UID: "73481f65-a2ac-45c1-be4e-eb29aa0915a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:12.940753 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.940733    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-token" pod "73481f65-a2ac-45c1-be4e-eb29aa0915a9" (UID: "73481f65-a2ac-45c1-be4e-eb29aa0915a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:12.940847 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:12.940748    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73481f65-a2ac-45c1-be4e-eb29aa0915a9-kube-api-access-dhmbs" pod "73481f65-a2ac-45c1-be4e-eb29aa0915a9" (UID: "73481f65-a2ac-45c1-be4e-eb29aa0915a9"). InnerVolumeSpecName "kube-api-access-dhmbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:13.040039 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:13.040002    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:13.040039 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:13.040031    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:13.040039 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:13.040044    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73481f65-a2ac-45c1-be4e-eb29aa0915a9-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:13.040257 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:13.040056    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhmbs\" (UniqueName: \"kubernetes.io/projected/73481f65-a2ac-45c1-be4e-eb29aa0915a9-kube-api-access-dhmbs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:13.049437 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:13.049404    4900 generic.go:356] "Generic (PLEG): container finished" podID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" containerID="ea601a4aa2c83cd331c7186d88c654921acdc374ae199454f1114902633dd41b" exitCode=0
Oct 31 03:59:13.049597 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:13.049458    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"
Oct 31 03:59:13.049597 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:13.049497    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" event={"ID":"73481f65-a2ac-45c1-be4e-eb29aa0915a9","Type":"ContainerDied","Data":"ea601a4aa2c83cd331c7186d88c654921acdc374ae199454f1114902633dd41b"}
Oct 31 03:59:13.049597 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:13.049561    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl" event={"ID":"73481f65-a2ac-45c1-be4e-eb29aa0915a9","Type":"ContainerDied","Data":"2d08dc31b406c5939dbd3148ad4bc7f317ad1fdcd4169d018ac71f816790cdc7"}
Oct 31 03:59:13.049597 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:13.049576    4900 scope.go:122] "RemoveContainer" containerID="ea601a4aa2c83cd331c7186d88c654921acdc374ae199454f1114902633dd41b"
Oct 31 03:59:13.062009 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:13.061988    4900 scope.go:122] "RemoveContainer" containerID="ea601a4aa2c83cd331c7186d88c654921acdc374ae199454f1114902633dd41b"
Oct 31 03:59:13.062278 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:59:13.062254    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea601a4aa2c83cd331c7186d88c654921acdc374ae199454f1114902633dd41b\": container with ID starting with ea601a4aa2c83cd331c7186d88c654921acdc374ae199454f1114902633dd41b not found: ID does not exist" containerID="ea601a4aa2c83cd331c7186d88c654921acdc374ae199454f1114902633dd41b"
Oct 31 03:59:13.062359 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:13.062287    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea601a4aa2c83cd331c7186d88c654921acdc374ae199454f1114902633dd41b"} err="failed to get container status \"ea601a4aa2c83cd331c7186d88c654921acdc374ae199454f1114902633dd41b\": rpc error: code = NotFound desc = could not find container \"ea601a4aa2c83cd331c7186d88c654921acdc374ae199454f1114902633dd41b\": container with ID starting with ea601a4aa2c83cd331c7186d88c654921acdc374ae199454f1114902633dd41b not found: ID does not exist"
Oct 31 03:59:13.215690 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:13.215616    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"]
Oct 31 03:59:13.217812 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:13.217779    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-bn7rl"]
Oct 31 03:59:15.153408 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:15.153377    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" path="/var/lib/kubelet/pods/73481f65-a2ac-45c1-be4e-eb29aa0915a9/volumes"
Oct 31 03:59:15.412929 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:15.412851    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:20.280332 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:20.280297    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:25.214946 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:25.214914    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:30.146363 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:30.146331    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:31.192242 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:31.192191    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:32.023208 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:32.023176    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:32.261454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:32.261417    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:35.379577 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:35.379544    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:36.815486 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:36.815451    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:36.988465 ip-10-0-134-83 kubenswrapper[4900]: E1031 03:59:36.988430    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 03:59:40.211474 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:40.211443    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:44.425472 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:44.425437    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:46.224016 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:46.223977    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:51.065696 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:51.065646    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:51.299152 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:51.299119    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:52.651960 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:52.651923    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:53.260201 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:53.260168    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:53.873769 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:53.873734    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:54.947934 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.947899    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"]
Oct 31 03:59:54.948405 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948392    4900 cpu_manager.go:414] "containerMap: removing container" podUID="eaea4507-660f-4b97-b8a7-d349c4441251" containerName="istio-proxy"
Oct 31 03:59:54.948454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948406    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaea4507-660f-4b97-b8a7-d349c4441251" containerName="istio-proxy"
Oct 31 03:59:54.948454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948436    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8e1017c9-1f08-4441-adfd-90415b9c6f17" containerName="discovery"
Oct 31 03:59:54.948454 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948441    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1017c9-1f08-4441-adfd-90415b9c6f17" containerName="discovery"
Oct 31 03:59:54.948552 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948455    4900 cpu_manager.go:414] "containerMap: removing container" podUID="7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" containerName="istio-proxy"
Oct 31 03:59:54.948552 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948461    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" containerName="istio-proxy"
Oct 31 03:59:54.948552 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948476    4900 cpu_manager.go:414] "containerMap: removing container" podUID="cf89487b-5877-4f0e-81f7-abb1ff44d51a" containerName="istio-proxy"
Oct 31 03:59:54.948552 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948480    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf89487b-5877-4f0e-81f7-abb1ff44d51a" containerName="istio-proxy"
Oct 31 03:59:54.948552 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948489    4900 cpu_manager.go:414] "containerMap: removing container" podUID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" containerName="istio-proxy"
Oct 31 03:59:54.948552 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948493    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" containerName="istio-proxy"
Oct 31 03:59:54.948744 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948564    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaea4507-660f-4b97-b8a7-d349c4441251" containerName="istio-proxy"
Oct 31 03:59:54.948744 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948576    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1017c9-1f08-4441-adfd-90415b9c6f17" containerName="discovery"
Oct 31 03:59:54.948744 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948584    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d719ccc-8bf1-4e14-a62e-0dcea593ed7a" containerName="istio-proxy"
Oct 31 03:59:54.948744 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948592    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf89487b-5877-4f0e-81f7-abb1ff44d51a" containerName="istio-proxy"
Oct 31 03:59:54.948744 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.948597    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="73481f65-a2ac-45c1-be4e-eb29aa0915a9" containerName="istio-proxy"
Oct 31 03:59:54.951969 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:54.951954    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.004854 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.004395    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-856ddc-openshift-default-dockercfg-ffnxl\""
Oct 31 03:59:55.015926 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.015900    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-envoy\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.016053 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.015935    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-workload-certs\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.016053 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.015962    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-credential-socket\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.016134 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.016068    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-token\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.016134 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.016090    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2323f996-abc1-4f51-9ed6-2e96cde226bc-istiod-ca-cert\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.016134 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.016107    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-workload-socket\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.016134 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.016123    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6gxw\" (UniqueName: \"kubernetes.io/projected/2323f996-abc1-4f51-9ed6-2e96cde226bc-kube-api-access-s6gxw\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.016425 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.016403    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-data\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.016470 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.016454    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-podinfo\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.058541 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.058509    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"]
Oct 31 03:59:55.099072 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.099042    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:55.118703 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.118674    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-data\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.118983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.118715    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-podinfo\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.118983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.118747    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-envoy\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.118983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.118765    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-workload-certs\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.118983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.118805    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-credential-socket\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.118983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.118857    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-token\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.118983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.118880    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2323f996-abc1-4f51-9ed6-2e96cde226bc-istiod-ca-cert\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.118983 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.118921    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-workload-socket\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.119543 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.119178    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-data\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.119543 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.119207    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-workload-certs\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.119543 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.119311    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-credential-socket\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.119543 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.119487    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-workload-socket\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.119689 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.119549    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2323f996-abc1-4f51-9ed6-2e96cde226bc-istiod-ca-cert\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.119689 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.119591    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6gxw\" (UniqueName: \"kubernetes.io/projected/2323f996-abc1-4f51-9ed6-2e96cde226bc-kube-api-access-s6gxw\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.121198 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.121097    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-envoy\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.121310 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.121222    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-podinfo\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.175702 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.175671    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-token\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.176358 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.176334    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6gxw\" (UniqueName: \"kubernetes.io/projected/2323f996-abc1-4f51-9ed6-2e96cde226bc-kube-api-access-s6gxw\") pod \"e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") " pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.264866 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.264833    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:55.331083 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.330950    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:55.685037 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.684948    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"]
Oct 31 03:59:55.716054 ip-10-0-134-83 kubenswrapper[4900]: W1031 03:59:55.716021    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2323f996_abc1_4f51_9ed6_2e96cde226bc.slice/crio-a51ea4d4bb8d5e8383fa7609922d0530c7c32deffe5e81106f2bffaf71b6b4ac WatchSource:0}: Error finding container a51ea4d4bb8d5e8383fa7609922d0530c7c32deffe5e81106f2bffaf71b6b4ac: Status 404 returned error can't find the container with id a51ea4d4bb8d5e8383fa7609922d0530c7c32deffe5e81106f2bffaf71b6b4ac
Oct 31 03:59:55.718599 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.718566    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:59:55.718700 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.718633    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:59:55.718700 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:55.718664    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:59:56.302496 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:56.302461    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" event={"ID":"2323f996-abc1-4f51-9ed6-2e96cde226bc","Type":"ContainerStarted","Data":"a2549ed5802638c525b9c572795964804d4acd67036fefe848f7920d34368a45"}
Oct 31 03:59:56.302941 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:56.302510    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" event={"ID":"2323f996-abc1-4f51-9ed6-2e96cde226bc","Type":"ContainerStarted","Data":"a51ea4d4bb8d5e8383fa7609922d0530c7c32deffe5e81106f2bffaf71b6b4ac"}
Oct 31 03:59:56.421319 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:56.421263    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" podStartSLOduration=2.421247697 podStartE2EDuration="2.421247697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:56.419915713 +0000 UTC m=+3149.386249068" watchObservedRunningTime="2026-10-31 03:59:56.421247697 +0000 UTC m=+3149.387581041"
Oct 31 03:59:56.609268 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:56.609181    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:57.265050 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:57.265011    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:57.271107 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:57.271078    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:57.308179 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:57.308140    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:57.309209 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:57.309186    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 03:59:59.334077 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:59.334046    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:59.560658 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:59.560623    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 03:59:59.956214 ip-10-0-134-83 kubenswrapper[4900]: I1031 03:59:59.956180    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:00.208085 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.208000    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"]
Oct 31 04:00:00.211904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.211886    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"]
Oct 31 04:00:00.212049 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.212024    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"
Oct 31 04:00:00.215646 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.215620    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"]
Oct 31 04:00:00.215769 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.215744    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"
Oct 31 04:00:00.228342 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.228315    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"
Oct 31 04:00:00.257971 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.257944    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"]
Oct 31 04:00:00.261919 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.261897    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"]
Oct 31 04:00:00.262056 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.262014    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"
Oct 31 04:00:00.265995 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.265968    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"
Oct 31 04:00:00.266297 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.266270    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:00:00.280760 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.280738    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8993f9fc-9cf4-4437-986c-1d43928ebd8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5\" (UID: \"8993f9fc-9cf4-4437-986c-1d43928ebd8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"
Oct 31 04:00:00.280995 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.280962    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg4d4\" (UniqueName: \"kubernetes.io/projected/1242b4c4-226f-4826-8900-656c58049ade-kube-api-access-vg4d4\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd\" (UID: \"1242b4c4-226f-4826-8900-656c58049ade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"
Oct 31 04:00:00.281173 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.281146    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldkc8\" (UniqueName: \"kubernetes.io/projected/8993f9fc-9cf4-4437-986c-1d43928ebd8b-kube-api-access-ldkc8\") pod \"maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5\" (UID: \"8993f9fc-9cf4-4437-986c-1d43928ebd8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"
Oct 31 04:00:00.281428 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.281409    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1242b4c4-226f-4826-8900-656c58049ade-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd\" (UID: \"1242b4c4-226f-4826-8900-656c58049ade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"
Oct 31 04:00:00.303827 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.303758    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"]
Oct 31 04:00:00.306085 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.306059    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"]
Oct 31 04:00:00.310688 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.310661    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"]
Oct 31 04:00:00.310910 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.310773    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"
Oct 31 04:00:00.311449 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.311412    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"]
Oct 31 04:00:00.328144 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.328109    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"]
Oct 31 04:00:00.329823 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.329363    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"]
Oct 31 04:00:00.348630 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.347611    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"]
Oct 31 04:00:00.382760 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.382720    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fcca104-5bd0-4498-b230-893bdcd090d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890320-9z86g\" (UID: \"5fcca104-5bd0-4498-b230-893bdcd090d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"
Oct 31 04:00:00.382760 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.382756    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1242b4c4-226f-4826-8900-656c58049ade-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd\" (UID: \"1242b4c4-226f-4826-8900-656c58049ade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"
Oct 31 04:00:00.383007 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.382782    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbksx\" (UniqueName: \"kubernetes.io/projected/ad1d8dfc-1271-4776-9af8-6429d8a201bf-kube-api-access-dbksx\") pod \"maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t\" (UID: \"ad1d8dfc-1271-4776-9af8-6429d8a201bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"
Oct 31 04:00:00.383007 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.382850    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8993f9fc-9cf4-4437-986c-1d43928ebd8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5\" (UID: \"8993f9fc-9cf4-4437-986c-1d43928ebd8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"
Oct 31 04:00:00.383007 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.382983    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg4d4\" (UniqueName: \"kubernetes.io/projected/1242b4c4-226f-4826-8900-656c58049ade-kube-api-access-vg4d4\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd\" (UID: \"1242b4c4-226f-4826-8900-656c58049ade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"
Oct 31 04:00:00.383140 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.383031    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1d8dfc-1271-4776-9af8-6429d8a201bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t\" (UID: \"ad1d8dfc-1271-4776-9af8-6429d8a201bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"
Oct 31 04:00:00.383140 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.383077    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b15a45-31af-4080-8597-edf79c2d3d8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v\" (UID: \"01b15a45-31af-4080-8597-edf79c2d3d8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"
Oct 31 04:00:00.383140 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.383105    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwmz5\" (UniqueName: \"kubernetes.io/projected/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085-kube-api-access-gwmz5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq\" (UID: \"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"
Oct 31 04:00:00.383307 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.383232    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldkc8\" (UniqueName: \"kubernetes.io/projected/8993f9fc-9cf4-4437-986c-1d43928ebd8b-kube-api-access-ldkc8\") pod \"maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5\" (UID: \"8993f9fc-9cf4-4437-986c-1d43928ebd8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"
Oct 31 04:00:00.383370 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.383342    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkhdb\" (UniqueName: \"kubernetes.io/projected/01b15a45-31af-4080-8597-edf79c2d3d8d-kube-api-access-hkhdb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v\" (UID: \"01b15a45-31af-4080-8597-edf79c2d3d8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"
Oct 31 04:00:00.383441 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.383422    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6fsx\" (UniqueName: \"kubernetes.io/projected/5fcca104-5bd0-4498-b230-893bdcd090d1-kube-api-access-d6fsx\") pod \"maas-api-key-cleanup-29890320-9z86g\" (UID: \"5fcca104-5bd0-4498-b230-893bdcd090d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"
Oct 31 04:00:00.383503 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.383456    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq\" (UID: \"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"
Oct 31 04:00:00.383671 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.383650    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8993f9fc-9cf4-4437-986c-1d43928ebd8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5\" (UID: \"8993f9fc-9cf4-4437-986c-1d43928ebd8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"
Oct 31 04:00:00.383757 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.383738    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1242b4c4-226f-4826-8900-656c58049ade-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd\" (UID: \"1242b4c4-226f-4826-8900-656c58049ade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"
Oct 31 04:00:00.440945 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.440917    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg4d4\" (UniqueName: \"kubernetes.io/projected/1242b4c4-226f-4826-8900-656c58049ade-kube-api-access-vg4d4\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd\" (UID: \"1242b4c4-226f-4826-8900-656c58049ade\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"
Oct 31 04:00:00.441097 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.441034    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldkc8\" (UniqueName: \"kubernetes.io/projected/8993f9fc-9cf4-4437-986c-1d43928ebd8b-kube-api-access-ldkc8\") pod \"maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5\" (UID: \"8993f9fc-9cf4-4437-986c-1d43928ebd8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"
Oct 31 04:00:00.484435 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.484400    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1d8dfc-1271-4776-9af8-6429d8a201bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t\" (UID: \"ad1d8dfc-1271-4776-9af8-6429d8a201bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"
Oct 31 04:00:00.484622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.484447    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b15a45-31af-4080-8597-edf79c2d3d8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v\" (UID: \"01b15a45-31af-4080-8597-edf79c2d3d8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"
Oct 31 04:00:00.484622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.484468    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwmz5\" (UniqueName: \"kubernetes.io/projected/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085-kube-api-access-gwmz5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq\" (UID: \"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"
Oct 31 04:00:00.484622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.484520    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkhdb\" (UniqueName: \"kubernetes.io/projected/01b15a45-31af-4080-8597-edf79c2d3d8d-kube-api-access-hkhdb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v\" (UID: \"01b15a45-31af-4080-8597-edf79c2d3d8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"
Oct 31 04:00:00.484622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.484544    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6fsx\" (UniqueName: \"kubernetes.io/projected/5fcca104-5bd0-4498-b230-893bdcd090d1-kube-api-access-d6fsx\") pod \"maas-api-key-cleanup-29890320-9z86g\" (UID: \"5fcca104-5bd0-4498-b230-893bdcd090d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"
Oct 31 04:00:00.484622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.484567    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq\" (UID: \"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"
Oct 31 04:00:00.485000 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.484682    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fcca104-5bd0-4498-b230-893bdcd090d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890320-9z86g\" (UID: \"5fcca104-5bd0-4498-b230-893bdcd090d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"
Oct 31 04:00:00.485000 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.484889    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbksx\" (UniqueName: \"kubernetes.io/projected/ad1d8dfc-1271-4776-9af8-6429d8a201bf-kube-api-access-dbksx\") pod \"maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t\" (UID: \"ad1d8dfc-1271-4776-9af8-6429d8a201bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"
Oct 31 04:00:00.485266 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.485243    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b15a45-31af-4080-8597-edf79c2d3d8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v\" (UID: \"01b15a45-31af-4080-8597-edf79c2d3d8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"
Oct 31 04:00:00.485359 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.485338    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq\" (UID: \"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"
Oct 31 04:00:00.485415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.485357    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1d8dfc-1271-4776-9af8-6429d8a201bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t\" (UID: \"ad1d8dfc-1271-4776-9af8-6429d8a201bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"
Oct 31 04:00:00.485415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.485374    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fcca104-5bd0-4498-b230-893bdcd090d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890320-9z86g\" (UID: \"5fcca104-5bd0-4498-b230-893bdcd090d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"
Oct 31 04:00:00.536707 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.536667    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"
Oct 31 04:00:00.541596 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.541489    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwmz5\" (UniqueName: \"kubernetes.io/projected/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085-kube-api-access-gwmz5\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq\" (UID: \"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"
Oct 31 04:00:00.541670 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.541600    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbksx\" (UniqueName: \"kubernetes.io/projected/ad1d8dfc-1271-4776-9af8-6429d8a201bf-kube-api-access-dbksx\") pod \"maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t\" (UID: \"ad1d8dfc-1271-4776-9af8-6429d8a201bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"
Oct 31 04:00:00.541670 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.541608    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkhdb\" (UniqueName: \"kubernetes.io/projected/01b15a45-31af-4080-8597-edf79c2d3d8d-kube-api-access-hkhdb\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v\" (UID: \"01b15a45-31af-4080-8597-edf79c2d3d8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"
Oct 31 04:00:00.542256 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.542230    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6fsx\" (UniqueName: \"kubernetes.io/projected/5fcca104-5bd0-4498-b230-893bdcd090d1-kube-api-access-d6fsx\") pod \"maas-api-key-cleanup-29890320-9z86g\" (UID: \"5fcca104-5bd0-4498-b230-893bdcd090d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"
Oct 31 04:00:00.544398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.544375    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"
Oct 31 04:00:00.549624 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.549603    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"
Oct 31 04:00:00.577943 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.577913    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"
Oct 31 04:00:00.589857 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.584851    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"
Oct 31 04:00:00.635850 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.635781    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"
Oct 31 04:00:00.921852 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:00:00.921821    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1242b4c4_226f_4826_8900_656c58049ade.slice/crio-625694e3a3cdce07722e07774101317425799204b5df3773ccb65c7bece490dd WatchSource:0}: Error finding container 625694e3a3cdce07722e07774101317425799204b5df3773ccb65c7bece490dd: Status 404 returned error can't find the container with id 625694e3a3cdce07722e07774101317425799204b5df3773ccb65c7bece490dd
Oct 31 04:00:00.924147 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.924121    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"]
Oct 31 04:00:00.968886 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:00:00.968844    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8993f9fc_9cf4_4437_986c_1d43928ebd8b.slice/crio-cb1a38df560147fdff115c50757a828e23ab0ae32a79aa987fa3ca6f1a4b21e7 WatchSource:0}: Error finding container cb1a38df560147fdff115c50757a828e23ab0ae32a79aa987fa3ca6f1a4b21e7: Status 404 returned error can't find the container with id cb1a38df560147fdff115c50757a828e23ab0ae32a79aa987fa3ca6f1a4b21e7
Oct 31 04:00:00.969869 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:00:00.969841    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1d8dfc_1271_4776_9af8_6429d8a201bf.slice/crio-f39c6ce0686e87032aa257104501c8bac07757795313afc511de3f91e5c4ead3 WatchSource:0}: Error finding container f39c6ce0686e87032aa257104501c8bac07757795313afc511de3f91e5c4ead3: Status 404 returned error can't find the container with id f39c6ce0686e87032aa257104501c8bac07757795313afc511de3f91e5c4ead3
Oct 31 04:00:00.971359 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.971340    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"]
Oct 31 04:00:00.974450 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:00.974432    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"]
Oct 31 04:00:01.019446 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:00:01.019423    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fcca104_5bd0_4498_b230_893bdcd090d1.slice/crio-148a590f5dc6424a06ab1c853c94de90f543ccc0eea38265b20a72a3259e939e WatchSource:0}: Error finding container 148a590f5dc6424a06ab1c853c94de90f543ccc0eea38265b20a72a3259e939e: Status 404 returned error can't find the container with id 148a590f5dc6424a06ab1c853c94de90f543ccc0eea38265b20a72a3259e939e
Oct 31 04:00:01.022320 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:01.022297    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"]
Oct 31 04:00:01.025281 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:00:01.025256    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b15a45_31af_4080_8597_edf79c2d3d8d.slice/crio-7fd41ef2caa93d55c1866ba902bdb64501a0639206b61d6a8a357b22a499108a WatchSource:0}: Error finding container 7fd41ef2caa93d55c1866ba902bdb64501a0639206b61d6a8a357b22a499108a: Status 404 returned error can't find the container with id 7fd41ef2caa93d55c1866ba902bdb64501a0639206b61d6a8a357b22a499108a
Oct 31 04:00:01.027118 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:01.027095    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"]
Oct 31 04:00:01.042868 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:01.042805    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"]
Oct 31 04:00:01.138218 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:00:01.138184    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef4fed3_5ba5_4af9_b3fb_9f3ae8b57085.slice/crio-b695d119fb88a4ea903b925f5e1a669311c97c6cccfbff8042789e46033d975e WatchSource:0}: Error finding container b695d119fb88a4ea903b925f5e1a669311c97c6cccfbff8042789e46033d975e: Status 404 returned error can't find the container with id b695d119fb88a4ea903b925f5e1a669311c97c6cccfbff8042789e46033d975e
Oct 31 04:00:01.343505 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:01.343467    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq" event={"ID":"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085","Type":"ContainerStarted","Data":"b695d119fb88a4ea903b925f5e1a669311c97c6cccfbff8042789e46033d975e"}
Oct 31 04:00:01.344912 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:01.344888    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t" event={"ID":"ad1d8dfc-1271-4776-9af8-6429d8a201bf","Type":"ContainerStarted","Data":"f39c6ce0686e87032aa257104501c8bac07757795313afc511de3f91e5c4ead3"}
Oct 31 04:00:01.346435 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:01.346403    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd" event={"ID":"1242b4c4-226f-4826-8900-656c58049ade","Type":"ContainerStarted","Data":"625694e3a3cdce07722e07774101317425799204b5df3773ccb65c7bece490dd"}
Oct 31 04:00:01.347848 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:01.347824    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v" event={"ID":"01b15a45-31af-4080-8597-edf79c2d3d8d","Type":"ContainerStarted","Data":"7fd41ef2caa93d55c1866ba902bdb64501a0639206b61d6a8a357b22a499108a"}
Oct 31 04:00:01.349450 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:01.349428    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g" event={"ID":"5fcca104-5bd0-4498-b230-893bdcd090d1","Type":"ContainerStarted","Data":"148a590f5dc6424a06ab1c853c94de90f543ccc0eea38265b20a72a3259e939e"}
Oct 31 04:00:01.351163 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:01.351138    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5" event={"ID":"8993f9fc-9cf4-4437-986c-1d43928ebd8b","Type":"ContainerStarted","Data":"cb1a38df560147fdff115c50757a828e23ab0ae32a79aa987fa3ca6f1a4b21e7"}
Oct 31 04:00:02.090094 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.090064    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:02.358846 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.358780    4900 generic.go:356] "Generic (PLEG): container finished" podID="01b15a45-31af-4080-8597-edf79c2d3d8d" containerID="9041113723f25e3b4c2cb2c4633bc2275beff7e21d09bfc7836b0e9dca921df3" exitCode=60
Oct 31 04:00:02.359292 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.358820    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v" event={"ID":"01b15a45-31af-4080-8597-edf79c2d3d8d","Type":"ContainerDied","Data":"9041113723f25e3b4c2cb2c4633bc2275beff7e21d09bfc7836b0e9dca921df3"}
Oct 31 04:00:02.359292 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.359170    4900 scope.go:122] "RemoveContainer" containerID="9041113723f25e3b4c2cb2c4633bc2275beff7e21d09bfc7836b0e9dca921df3"
Oct 31 04:00:02.360628 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.360605    4900 generic.go:356] "Generic (PLEG): container finished" podID="5fcca104-5bd0-4498-b230-893bdcd090d1" containerID="65d9f0f456141d9245f9f65729a906fa7fdb95ef49b460d2ebb5953c8eae3a0b" exitCode=60
Oct 31 04:00:02.360754 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.360691    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g" event={"ID":"5fcca104-5bd0-4498-b230-893bdcd090d1","Type":"ContainerDied","Data":"65d9f0f456141d9245f9f65729a906fa7fdb95ef49b460d2ebb5953c8eae3a0b"}
Oct 31 04:00:02.360946 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.360937    4900 scope.go:122] "RemoveContainer" containerID="65d9f0f456141d9245f9f65729a906fa7fdb95ef49b460d2ebb5953c8eae3a0b"
Oct 31 04:00:02.362416 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.362396    4900 generic.go:356] "Generic (PLEG): container finished" podID="8993f9fc-9cf4-4437-986c-1d43928ebd8b" containerID="d541eaed280010462a3a582912cda102899c52c67f043562895798c4013686dd" exitCode=60
Oct 31 04:00:02.362485 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.362424    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5" event={"ID":"8993f9fc-9cf4-4437-986c-1d43928ebd8b","Type":"ContainerDied","Data":"d541eaed280010462a3a582912cda102899c52c67f043562895798c4013686dd"}
Oct 31 04:00:02.362688 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.362666    4900 scope.go:122] "RemoveContainer" containerID="d541eaed280010462a3a582912cda102899c52c67f043562895798c4013686dd"
Oct 31 04:00:02.364003 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.363969    4900 generic.go:356] "Generic (PLEG): container finished" podID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" containerID="5fb45ec9f700ac394b94c2c9298692a963dcea220fb0f031ebe776eb27e73701" exitCode=60
Oct 31 04:00:02.364071 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.364045    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq" event={"ID":"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085","Type":"ContainerDied","Data":"5fb45ec9f700ac394b94c2c9298692a963dcea220fb0f031ebe776eb27e73701"}
Oct 31 04:00:02.364234 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.364217    4900 scope.go:122] "RemoveContainer" containerID="5fb45ec9f700ac394b94c2c9298692a963dcea220fb0f031ebe776eb27e73701"
Oct 31 04:00:02.365502 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.365483    4900 generic.go:356] "Generic (PLEG): container finished" podID="ad1d8dfc-1271-4776-9af8-6429d8a201bf" containerID="4a9b105e4c64c502ce6ef1c7fd313ff8f025677e56e64974bb79f85ac091dc25" exitCode=60
Oct 31 04:00:02.365592 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.365507    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t" event={"ID":"ad1d8dfc-1271-4776-9af8-6429d8a201bf","Type":"ContainerDied","Data":"4a9b105e4c64c502ce6ef1c7fd313ff8f025677e56e64974bb79f85ac091dc25"}
Oct 31 04:00:02.365782 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.365763    4900 scope.go:122] "RemoveContainer" containerID="4a9b105e4c64c502ce6ef1c7fd313ff8f025677e56e64974bb79f85ac091dc25"
Oct 31 04:00:02.367002 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.366980    4900 generic.go:356] "Generic (PLEG): container finished" podID="1242b4c4-226f-4826-8900-656c58049ade" containerID="2db82ee4cdcbcff6a08ef9e62eafbf31be9c76ab49e3dd56750c14bf50c39ea1" exitCode=60
Oct 31 04:00:02.367101 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.367035    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd" event={"ID":"1242b4c4-226f-4826-8900-656c58049ade","Type":"ContainerDied","Data":"2db82ee4cdcbcff6a08ef9e62eafbf31be9c76ab49e3dd56750c14bf50c39ea1"}
Oct 31 04:00:02.367239 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.367223    4900 scope.go:122] "RemoveContainer" containerID="2db82ee4cdcbcff6a08ef9e62eafbf31be9c76ab49e3dd56750c14bf50c39ea1"
Oct 31 04:00:02.812113 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:02.812082    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:03.057497 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.057418    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:03.138911 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.138875    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"]
Oct 31 04:00:03.143034 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.143012    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"
Oct 31 04:00:03.193889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.193854    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"]
Oct 31 04:00:03.195282 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.195253    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-856ddc-dockercfg-zqnjs\""
Oct 31 04:00:03.198007 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.197985    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"]
Oct 31 04:00:03.198174 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.198097    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"
Oct 31 04:00:03.208410 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.208384    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-856ddc\""
Oct 31 04:00:03.270840 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.269888    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"]
Oct 31 04:00:03.327019 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.326933    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5420545a-5a77-4a15-bea3-8274d49e5e5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x\" (UID: \"5420545a-5a77-4a15-bea3-8274d49e5e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"
Oct 31 04:00:03.327170 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.327094    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ctvh\" (UniqueName: \"kubernetes.io/projected/5420545a-5a77-4a15-bea3-8274d49e5e5d-kube-api-access-7ctvh\") pod \"payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x\" (UID: \"5420545a-5a77-4a15-bea3-8274d49e5e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"
Oct 31 04:00:03.327297 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.327279    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319c12fa-ccdd-4dac-83e5-802273cc62bf-plugins-config-volume\") pod \"payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c\" (UID: \"319c12fa-ccdd-4dac-83e5-802273cc62bf\") " pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"
Oct 31 04:00:03.327368 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.327329    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnbkm\" (UniqueName: \"kubernetes.io/projected/319c12fa-ccdd-4dac-83e5-802273cc62bf-kube-api-access-rnbkm\") pod \"payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c\" (UID: \"319c12fa-ccdd-4dac-83e5-802273cc62bf\") " pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"
Oct 31 04:00:03.374393 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.374358    4900 generic.go:356] "Generic (PLEG): container finished" podID="01b15a45-31af-4080-8597-edf79c2d3d8d" containerID="b073d3290238887cfb7ac4f12207620d4abf02b200964f126738411557382b89" exitCode=60
Oct 31 04:00:03.374831 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.374439    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v" event={"ID":"01b15a45-31af-4080-8597-edf79c2d3d8d","Type":"ContainerDied","Data":"b073d3290238887cfb7ac4f12207620d4abf02b200964f126738411557382b89"}
Oct 31 04:00:03.374831 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.374515    4900 scope.go:122] "RemoveContainer" containerID="9041113723f25e3b4c2cb2c4633bc2275beff7e21d09bfc7836b0e9dca921df3"
Oct 31 04:00:03.374956 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.374935    4900 scope.go:122] "RemoveContainer" containerID="b073d3290238887cfb7ac4f12207620d4abf02b200964f126738411557382b89"
Oct 31 04:00:03.375272 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:03.375252    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v_odh-ai-gateway-infra(01b15a45-31af-4080-8597-edf79c2d3d8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v" podUID="01b15a45-31af-4080-8597-edf79c2d3d8d"
Oct 31 04:00:03.376922 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.376898    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g" event={"ID":"5fcca104-5bd0-4498-b230-893bdcd090d1","Type":"ContainerStarted","Data":"6598bcd12abc6da187c9a17e4c94c9d58abb962e12b994dd112e4246807af478"}
Oct 31 04:00:03.378783 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.378762    4900 generic.go:356] "Generic (PLEG): container finished" podID="8993f9fc-9cf4-4437-986c-1d43928ebd8b" containerID="c3ef3bec4691aa4e3b62750687822c82fd22f6076acd06a91105ee84d22f594d" exitCode=60
Oct 31 04:00:03.378928 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.378894    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5" event={"ID":"8993f9fc-9cf4-4437-986c-1d43928ebd8b","Type":"ContainerDied","Data":"c3ef3bec4691aa4e3b62750687822c82fd22f6076acd06a91105ee84d22f594d"}
Oct 31 04:00:03.379140 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.379116    4900 scope.go:122] "RemoveContainer" containerID="c3ef3bec4691aa4e3b62750687822c82fd22f6076acd06a91105ee84d22f594d"
Oct 31 04:00:03.379466 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:03.379432    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5_odh-ai-gateway-infra(8993f9fc-9cf4-4437-986c-1d43928ebd8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5" podUID="8993f9fc-9cf4-4437-986c-1d43928ebd8b"
Oct 31 04:00:03.380821 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.380797    4900 generic.go:356] "Generic (PLEG): container finished" podID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" containerID="75f135734d1696f538f37942d813b8cc6be09744422ac9f7c1035b5c7ccdc7fc" exitCode=60
Oct 31 04:00:03.380921 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.380869    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq" event={"ID":"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085","Type":"ContainerDied","Data":"75f135734d1696f538f37942d813b8cc6be09744422ac9f7c1035b5c7ccdc7fc"}
Oct 31 04:00:03.381151 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.381130    4900 scope.go:122] "RemoveContainer" containerID="75f135734d1696f538f37942d813b8cc6be09744422ac9f7c1035b5c7ccdc7fc"
Oct 31 04:00:03.381411 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:03.381391    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq_odh-ai-gateway-infra(1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq" podUID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085"
Oct 31 04:00:03.382752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.382732    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t" event={"ID":"ad1d8dfc-1271-4776-9af8-6429d8a201bf","Type":"ContainerStarted","Data":"a06cbea8ff00cd76b844b827178b34c09e1b2a361faebc3b288292e33d7523e1"}
Oct 31 04:00:03.384398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.384378    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd" event={"ID":"1242b4c4-226f-4826-8900-656c58049ade","Type":"ContainerStarted","Data":"c5f86c35881264045260acac46eaff6e0aba5cd78def39a12c86baa3313609d7"}
Oct 31 04:00:03.428453 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.428418    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ctvh\" (UniqueName: \"kubernetes.io/projected/5420545a-5a77-4a15-bea3-8274d49e5e5d-kube-api-access-7ctvh\") pod \"payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x\" (UID: \"5420545a-5a77-4a15-bea3-8274d49e5e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"
Oct 31 04:00:03.428590 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.428531    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319c12fa-ccdd-4dac-83e5-802273cc62bf-plugins-config-volume\") pod \"payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c\" (UID: \"319c12fa-ccdd-4dac-83e5-802273cc62bf\") " pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"
Oct 31 04:00:03.428590 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.428560    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnbkm\" (UniqueName: \"kubernetes.io/projected/319c12fa-ccdd-4dac-83e5-802273cc62bf-kube-api-access-rnbkm\") pod \"payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c\" (UID: \"319c12fa-ccdd-4dac-83e5-802273cc62bf\") " pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"
Oct 31 04:00:03.428721 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.428703    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5420545a-5a77-4a15-bea3-8274d49e5e5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x\" (UID: \"5420545a-5a77-4a15-bea3-8274d49e5e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"
Oct 31 04:00:03.429284 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.429261    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319c12fa-ccdd-4dac-83e5-802273cc62bf-plugins-config-volume\") pod \"payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c\" (UID: \"319c12fa-ccdd-4dac-83e5-802273cc62bf\") " pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"
Oct 31 04:00:03.429430 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.429408    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5420545a-5a77-4a15-bea3-8274d49e5e5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x\" (UID: \"5420545a-5a77-4a15-bea3-8274d49e5e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"
Oct 31 04:00:03.488358 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.488324    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ctvh\" (UniqueName: \"kubernetes.io/projected/5420545a-5a77-4a15-bea3-8274d49e5e5d-kube-api-access-7ctvh\") pod \"payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x\" (UID: \"5420545a-5a77-4a15-bea3-8274d49e5e5d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"
Oct 31 04:00:03.488481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.488324    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnbkm\" (UniqueName: \"kubernetes.io/projected/319c12fa-ccdd-4dac-83e5-802273cc62bf-kube-api-access-rnbkm\") pod \"payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c\" (UID: \"319c12fa-ccdd-4dac-83e5-802273cc62bf\") " pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"
Oct 31 04:00:03.513498 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.513471    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"
Oct 31 04:00:03.608175 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.608117    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g" podStartSLOduration=3.608098918 podStartE2EDuration="3.608098918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:03.606309155 +0000 UTC m=+3156.572642504" watchObservedRunningTime="2026-10-31 04:00:03.608098918 +0000 UTC m=+3156.574432263"
Oct 31 04:00:03.668856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.668821    4900 scope.go:122] "RemoveContainer" containerID="d541eaed280010462a3a582912cda102899c52c67f043562895798c4013686dd"
Oct 31 04:00:03.681690 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.681669    4900 scope.go:122] "RemoveContainer" containerID="5fb45ec9f700ac394b94c2c9298692a963dcea220fb0f031ebe776eb27e73701"
Oct 31 04:00:03.760688 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.760652    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"
Oct 31 04:00:03.835639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.835531    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd" podStartSLOduration=3.835510949 podStartE2EDuration="3.835510949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:03.833440726 +0000 UTC m=+3156.799774072" watchObservedRunningTime="2026-10-31 04:00:03.835510949 +0000 UTC m=+3156.801844294"
Oct 31 04:00:03.880810 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:00:03.880755    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319c12fa_ccdd_4dac_83e5_802273cc62bf.slice/crio-9914fc397eb881303b98cd6db0ec68513c5b85f3bc44f870764a97af4522b967 WatchSource:0}: Error finding container 9914fc397eb881303b98cd6db0ec68513c5b85f3bc44f870764a97af4522b967: Status 404 returned error can't find the container with id 9914fc397eb881303b98cd6db0ec68513c5b85f3bc44f870764a97af4522b967
Oct 31 04:00:03.883138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:03.883112    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"]
Oct 31 04:00:04.073435 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.073383    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t" podStartSLOduration=4.073369628 podStartE2EDuration="4.073369628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:04.07204742 +0000 UTC m=+3157.038380764" watchObservedRunningTime="2026-10-31 04:00:04.073369628 +0000 UTC m=+3157.039702972"
Oct 31 04:00:04.132493 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:00:04.132456    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5420545a_5a77_4a15_bea3_8274d49e5e5d.slice/crio-b917888cabab809d24e8825a0dc44936cecbd3ec350b410ab9525cbc14d8f849 WatchSource:0}: Error finding container b917888cabab809d24e8825a0dc44936cecbd3ec350b410ab9525cbc14d8f849: Status 404 returned error can't find the container with id b917888cabab809d24e8825a0dc44936cecbd3ec350b410ab9525cbc14d8f849
Oct 31 04:00:04.134976 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.134951    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"]
Oct 31 04:00:04.395728 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.395625    4900 generic.go:356] "Generic (PLEG): container finished" podID="1242b4c4-226f-4826-8900-656c58049ade" containerID="c5f86c35881264045260acac46eaff6e0aba5cd78def39a12c86baa3313609d7" exitCode=60
Oct 31 04:00:04.395728 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.395711    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd" event={"ID":"1242b4c4-226f-4826-8900-656c58049ade","Type":"ContainerDied","Data":"c5f86c35881264045260acac46eaff6e0aba5cd78def39a12c86baa3313609d7"}
Oct 31 04:00:04.396268 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.395816    4900 scope.go:122] "RemoveContainer" containerID="2db82ee4cdcbcff6a08ef9e62eafbf31be9c76ab49e3dd56750c14bf50c39ea1"
Oct 31 04:00:04.396268 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.396052    4900 scope.go:122] "RemoveContainer" containerID="c5f86c35881264045260acac46eaff6e0aba5cd78def39a12c86baa3313609d7"
Oct 31 04:00:04.396436 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:04.396412    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd_odh-ai-gateway-infra(1242b4c4-226f-4826-8900-656c58049ade)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd" podUID="1242b4c4-226f-4826-8900-656c58049ade"
Oct 31 04:00:04.397830 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.397811    4900 scope.go:122] "RemoveContainer" containerID="b073d3290238887cfb7ac4f12207620d4abf02b200964f126738411557382b89"
Oct 31 04:00:04.398121 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:04.398099    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v_odh-ai-gateway-infra(01b15a45-31af-4080-8597-edf79c2d3d8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v" podUID="01b15a45-31af-4080-8597-edf79c2d3d8d"
Oct 31 04:00:04.399014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.398987    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c" event={"ID":"319c12fa-ccdd-4dac-83e5-802273cc62bf","Type":"ContainerStarted","Data":"41d3e3e464977f3aec61abab5a99417f732da6e032c6e6758683e7b1fd6e2e3a"}
Oct 31 04:00:04.399116 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.399042    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"
Oct 31 04:00:04.399116 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.399067    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c" event={"ID":"319c12fa-ccdd-4dac-83e5-802273cc62bf","Type":"ContainerStarted","Data":"9914fc397eb881303b98cd6db0ec68513c5b85f3bc44f870764a97af4522b967"}
Oct 31 04:00:04.400494 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.400474    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x" event={"ID":"5420545a-5a77-4a15-bea3-8274d49e5e5d","Type":"ContainerStarted","Data":"da24f71e3cc8c08bf687036e910e8c0fcd308e654ae3c71876b0cd52dc3e1216"}
Oct 31 04:00:04.400582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.400511    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x" event={"ID":"5420545a-5a77-4a15-bea3-8274d49e5e5d","Type":"ContainerStarted","Data":"b917888cabab809d24e8825a0dc44936cecbd3ec350b410ab9525cbc14d8f849"}
Oct 31 04:00:04.400582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.400535    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"
Oct 31 04:00:04.402314 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.402296    4900 generic.go:356] "Generic (PLEG): container finished" podID="5fcca104-5bd0-4498-b230-893bdcd090d1" containerID="6598bcd12abc6da187c9a17e4c94c9d58abb962e12b994dd112e4246807af478" exitCode=60
Oct 31 04:00:04.402412 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.402362    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g" event={"ID":"5fcca104-5bd0-4498-b230-893bdcd090d1","Type":"ContainerDied","Data":"6598bcd12abc6da187c9a17e4c94c9d58abb962e12b994dd112e4246807af478"}
Oct 31 04:00:04.402602 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.402585    4900 scope.go:122] "RemoveContainer" containerID="6598bcd12abc6da187c9a17e4c94c9d58abb962e12b994dd112e4246807af478"
Oct 31 04:00:04.402873 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:04.402844    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890320-9z86g_odh-ai-gateway-infra(5fcca104-5bd0-4498-b230-893bdcd090d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g" podUID="5fcca104-5bd0-4498-b230-893bdcd090d1"
Oct 31 04:00:04.404119 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.404099    4900 scope.go:122] "RemoveContainer" containerID="c3ef3bec4691aa4e3b62750687822c82fd22f6076acd06a91105ee84d22f594d"
Oct 31 04:00:04.404371 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:04.404348    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5_odh-ai-gateway-infra(8993f9fc-9cf4-4437-986c-1d43928ebd8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5" podUID="8993f9fc-9cf4-4437-986c-1d43928ebd8b"
Oct 31 04:00:04.405837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.405815    4900 scope.go:122] "RemoveContainer" containerID="75f135734d1696f538f37942d813b8cc6be09744422ac9f7c1035b5c7ccdc7fc"
Oct 31 04:00:04.406103 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:04.406079    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq_odh-ai-gateway-infra(1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq" podUID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085"
Oct 31 04:00:04.407505 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.407475    4900 generic.go:356] "Generic (PLEG): container finished" podID="ad1d8dfc-1271-4776-9af8-6429d8a201bf" containerID="a06cbea8ff00cd76b844b827178b34c09e1b2a361faebc3b288292e33d7523e1" exitCode=60
Oct 31 04:00:04.407586 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.407509    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t" event={"ID":"ad1d8dfc-1271-4776-9af8-6429d8a201bf","Type":"ContainerDied","Data":"a06cbea8ff00cd76b844b827178b34c09e1b2a361faebc3b288292e33d7523e1"}
Oct 31 04:00:04.407713 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.407700    4900 scope.go:122] "RemoveContainer" containerID="a06cbea8ff00cd76b844b827178b34c09e1b2a361faebc3b288292e33d7523e1"
Oct 31 04:00:04.407913 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:04.407897    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t_odh-ai-gateway-infra(ad1d8dfc-1271-4776-9af8-6429d8a201bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t" podUID="ad1d8dfc-1271-4776-9af8-6429d8a201bf"
Oct 31 04:00:04.409130 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.409111    4900 scope.go:122] "RemoveContainer" containerID="65d9f0f456141d9245f9f65729a906fa7fdb95ef49b460d2ebb5953c8eae3a0b"
Oct 31 04:00:04.421981 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.421960    4900 scope.go:122] "RemoveContainer" containerID="4a9b105e4c64c502ce6ef1c7fd313ff8f025677e56e64974bb79f85ac091dc25"
Oct 31 04:00:04.851107 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:04.851056    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c" podStartSLOduration=1.851041469 podStartE2EDuration="1.851041469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:04.849938156 +0000 UTC m=+3157.816271500" watchObservedRunningTime="2026-10-31 04:00:04.851041469 +0000 UTC m=+3157.817374814"
Oct 31 04:00:05.302395 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:05.302336    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x" podStartSLOduration=2.302317618 podStartE2EDuration="2.302317618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:05.300768209 +0000 UTC m=+3158.267101552" watchObservedRunningTime="2026-10-31 04:00:05.302317618 +0000 UTC m=+3158.268650965"
Oct 31 04:00:05.417135 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:05.417102    4900 scope.go:122] "RemoveContainer" containerID="c5f86c35881264045260acac46eaff6e0aba5cd78def39a12c86baa3313609d7"
Oct 31 04:00:05.417646 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:05.417485    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd_odh-ai-gateway-infra(1242b4c4-226f-4826-8900-656c58049ade)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd" podUID="1242b4c4-226f-4826-8900-656c58049ade"
Oct 31 04:00:05.422730 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:05.422703    4900 scope.go:122] "RemoveContainer" containerID="6598bcd12abc6da187c9a17e4c94c9d58abb962e12b994dd112e4246807af478"
Oct 31 04:00:05.423125 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:05.423090    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890320-9z86g_odh-ai-gateway-infra(5fcca104-5bd0-4498-b230-893bdcd090d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g" podUID="5fcca104-5bd0-4498-b230-893bdcd090d1"
Oct 31 04:00:05.425680 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:05.425657    4900 scope.go:122] "RemoveContainer" containerID="a06cbea8ff00cd76b844b827178b34c09e1b2a361faebc3b288292e33d7523e1"
Oct 31 04:00:05.425980 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:05.425960    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t_odh-ai-gateway-infra(ad1d8dfc-1271-4776-9af8-6429d8a201bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t" podUID="ad1d8dfc-1271-4776-9af8-6429d8a201bf"
Oct 31 04:00:06.985217 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:06.985180    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:00:07.414125 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:07.414009    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:12.284200 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:12.284104    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:14.146064 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:14.146031    4900 scope.go:122] "RemoveContainer" containerID="6598bcd12abc6da187c9a17e4c94c9d58abb962e12b994dd112e4246807af478"
Oct 31 04:00:14.478476 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:14.478435    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g" event={"ID":"5fcca104-5bd0-4498-b230-893bdcd090d1","Type":"ContainerStarted","Data":"c78f223efc0d363fff9406e9657ef6aefdfdc24da9cd6e1b1c95e7eea6aec9e5"}
Oct 31 04:00:15.145633 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:15.145602    4900 scope.go:122] "RemoveContainer" containerID="c5f86c35881264045260acac46eaff6e0aba5cd78def39a12c86baa3313609d7"
Oct 31 04:00:15.145837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:15.145745    4900 scope.go:122] "RemoveContainer" containerID="c3ef3bec4691aa4e3b62750687822c82fd22f6076acd06a91105ee84d22f594d"
Oct 31 04:00:15.428506 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:15.428476    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"
Oct 31 04:00:15.429009 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:15.428550    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"
Oct 31 04:00:15.486014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:15.485979    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5" event={"ID":"8993f9fc-9cf4-4437-986c-1d43928ebd8b","Type":"ContainerStarted","Data":"9e0ebb06091f7e36d3cff2d38c2d11846781a2c71b686f164a5733979a59f09e"}
Oct 31 04:00:15.487832 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:15.487806    4900 generic.go:356] "Generic (PLEG): container finished" podID="5fcca104-5bd0-4498-b230-893bdcd090d1" containerID="c78f223efc0d363fff9406e9657ef6aefdfdc24da9cd6e1b1c95e7eea6aec9e5" exitCode=60
Oct 31 04:00:15.487954 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:15.487843    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g" event={"ID":"5fcca104-5bd0-4498-b230-893bdcd090d1","Type":"ContainerDied","Data":"c78f223efc0d363fff9406e9657ef6aefdfdc24da9cd6e1b1c95e7eea6aec9e5"}
Oct 31 04:00:15.487954 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:15.487892    4900 scope.go:122] "RemoveContainer" containerID="6598bcd12abc6da187c9a17e4c94c9d58abb962e12b994dd112e4246807af478"
Oct 31 04:00:15.488231 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:15.488216    4900 scope.go:122] "RemoveContainer" containerID="c78f223efc0d363fff9406e9657ef6aefdfdc24da9cd6e1b1c95e7eea6aec9e5"
Oct 31 04:00:15.488465 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:15.488441    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890320-9z86g_odh-ai-gateway-infra(5fcca104-5bd0-4498-b230-893bdcd090d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g" podUID="5fcca104-5bd0-4498-b230-893bdcd090d1"
Oct 31 04:00:15.602342 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:15.602307    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"]
Oct 31 04:00:15.769673 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:15.769619    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5" podStartSLOduration=15.769603718 podStartE2EDuration="15.769603718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:15.767966882 +0000 UTC m=+3168.734300226" watchObservedRunningTime="2026-10-31 04:00:15.769603718 +0000 UTC m=+3168.735937095"
Oct 31 04:00:16.152679 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.152645    4900 scope.go:122] "RemoveContainer" containerID="a06cbea8ff00cd76b844b827178b34c09e1b2a361faebc3b288292e33d7523e1"
Oct 31 04:00:16.201918 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.201877    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:16.497446 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.497418    4900 generic.go:356] "Generic (PLEG): container finished" podID="8993f9fc-9cf4-4437-986c-1d43928ebd8b" containerID="9e0ebb06091f7e36d3cff2d38c2d11846781a2c71b686f164a5733979a59f09e" exitCode=60
Oct 31 04:00:16.497837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.497489    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5" event={"ID":"8993f9fc-9cf4-4437-986c-1d43928ebd8b","Type":"ContainerDied","Data":"9e0ebb06091f7e36d3cff2d38c2d11846781a2c71b686f164a5733979a59f09e"}
Oct 31 04:00:16.497837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.497542    4900 scope.go:122] "RemoveContainer" containerID="c3ef3bec4691aa4e3b62750687822c82fd22f6076acd06a91105ee84d22f594d"
Oct 31 04:00:16.497953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.497871    4900 scope.go:122] "RemoveContainer" containerID="9e0ebb06091f7e36d3cff2d38c2d11846781a2c71b686f164a5733979a59f09e"
Oct 31 04:00:16.498184 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:16.498165    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5_odh-ai-gateway-infra(8993f9fc-9cf4-4437-986c-1d43928ebd8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5" podUID="8993f9fc-9cf4-4437-986c-1d43928ebd8b"
Oct 31 04:00:16.499977 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.499958    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t" event={"ID":"ad1d8dfc-1271-4776-9af8-6429d8a201bf","Type":"ContainerStarted","Data":"4163e60052c7df3c241c5a62c8398d5eddb3f51a8bf5f9897394c5f18181d2c9"}
Oct 31 04:00:16.501717 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.501693    4900 generic.go:356] "Generic (PLEG): container finished" podID="1242b4c4-226f-4826-8900-656c58049ade" containerID="47384cc9c035bf67b0e0f5c9695a91b08809355aff1f354c4ffddc29694b9205" exitCode=60
Oct 31 04:00:16.501837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.501756    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd" event={"ID":"1242b4c4-226f-4826-8900-656c58049ade","Type":"ContainerDied","Data":"47384cc9c035bf67b0e0f5c9695a91b08809355aff1f354c4ffddc29694b9205"}
Oct 31 04:00:16.502073 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.502059    4900 scope.go:122] "RemoveContainer" containerID="47384cc9c035bf67b0e0f5c9695a91b08809355aff1f354c4ffddc29694b9205"
Oct 31 04:00:16.502300 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:16.502271    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd_odh-ai-gateway-infra(1242b4c4-226f-4826-8900-656c58049ade)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd" podUID="1242b4c4-226f-4826-8900-656c58049ade"
Oct 31 04:00:16.505619 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.505581    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:16.511751 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.511731    4900 scope.go:122] "RemoveContainer" containerID="c5f86c35881264045260acac46eaff6e0aba5cd78def39a12c86baa3313609d7"
Oct 31 04:00:16.657484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.657461    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"
Oct 31 04:00:16.692031 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.692006    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fcca104-5bd0-4498-b230-893bdcd090d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fcca104-5bd0-4498-b230-893bdcd090d1-maas-api-service-ca\") pod \"5fcca104-5bd0-4498-b230-893bdcd090d1\" (UID: \"5fcca104-5bd0-4498-b230-893bdcd090d1\") "
Oct 31 04:00:16.692191 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.692132    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcca104-5bd0-4498-b230-893bdcd090d1-kube-api-access-d6fsx\" (UniqueName: \"kubernetes.io/projected/5fcca104-5bd0-4498-b230-893bdcd090d1-kube-api-access-d6fsx\") pod \"5fcca104-5bd0-4498-b230-893bdcd090d1\" (UID: \"5fcca104-5bd0-4498-b230-893bdcd090d1\") "
Oct 31 04:00:16.692327 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.692307    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fcca104-5bd0-4498-b230-893bdcd090d1-maas-api-service-ca" pod "5fcca104-5bd0-4498-b230-893bdcd090d1" (UID: "5fcca104-5bd0-4498-b230-893bdcd090d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:16.692508 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.692490    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fcca104-5bd0-4498-b230-893bdcd090d1-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:16.694043 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.694024    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcca104-5bd0-4498-b230-893bdcd090d1-kube-api-access-d6fsx" pod "5fcca104-5bd0-4498-b230-893bdcd090d1" (UID: "5fcca104-5bd0-4498-b230-893bdcd090d1"). InnerVolumeSpecName "kube-api-access-d6fsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:16.778379 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.778346    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"]
Oct 31 04:00:16.793343 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:16.793314    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6fsx\" (UniqueName: \"kubernetes.io/projected/5fcca104-5bd0-4498-b230-893bdcd090d1-kube-api-access-d6fsx\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:17.150239 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.150210    4900 scope.go:122] "RemoveContainer" containerID="b073d3290238887cfb7ac4f12207620d4abf02b200964f126738411557382b89"
Oct 31 04:00:17.150407 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.150359    4900 scope.go:122] "RemoveContainer" containerID="75f135734d1696f538f37942d813b8cc6be09744422ac9f7c1035b5c7ccdc7fc"
Oct 31 04:00:17.216973 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.216946    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:17.511491 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.511455    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v" event={"ID":"01b15a45-31af-4080-8597-edf79c2d3d8d","Type":"ContainerStarted","Data":"06fb92bff20900c3e9e618be53d9c86e0e598d31244407b2ea342d1fbada8c6d"}
Oct 31 04:00:17.512905 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.512885    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"
Oct 31 04:00:17.513044 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.512904    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g" event={"ID":"5fcca104-5bd0-4498-b230-893bdcd090d1","Type":"ContainerDied","Data":"148a590f5dc6424a06ab1c853c94de90f543ccc0eea38265b20a72a3259e939e"}
Oct 31 04:00:17.513044 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.512966    4900 scope.go:122] "RemoveContainer" containerID="c78f223efc0d363fff9406e9657ef6aefdfdc24da9cd6e1b1c95e7eea6aec9e5"
Oct 31 04:00:17.516919 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.516892    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq" event={"ID":"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085","Type":"ContainerStarted","Data":"d01e425a4c9ae0be13d5e883d5bceaf01fb8f28ff2712d8f8cfcca3d8f295c1c"}
Oct 31 04:00:17.518906 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.518879    4900 generic.go:356] "Generic (PLEG): container finished" podID="ad1d8dfc-1271-4776-9af8-6429d8a201bf" containerID="4163e60052c7df3c241c5a62c8398d5eddb3f51a8bf5f9897394c5f18181d2c9" exitCode=60
Oct 31 04:00:17.518994 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.518956    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t" event={"ID":"ad1d8dfc-1271-4776-9af8-6429d8a201bf","Type":"ContainerDied","Data":"4163e60052c7df3c241c5a62c8398d5eddb3f51a8bf5f9897394c5f18181d2c9"}
Oct 31 04:00:17.519301 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.519284    4900 scope.go:122] "RemoveContainer" containerID="4163e60052c7df3c241c5a62c8398d5eddb3f51a8bf5f9897394c5f18181d2c9"
Oct 31 04:00:17.519570 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:17.519550    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t_odh-ai-gateway-infra(ad1d8dfc-1271-4776-9af8-6429d8a201bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t" podUID="ad1d8dfc-1271-4776-9af8-6429d8a201bf"
Oct 31 04:00:17.526948 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.526929    4900 scope.go:122] "RemoveContainer" containerID="a06cbea8ff00cd76b844b827178b34c09e1b2a361faebc3b288292e33d7523e1"
Oct 31 04:00:17.631077 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.631029    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v" podStartSLOduration=17.631016086 podStartE2EDuration="17.631016086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:17.630185401 +0000 UTC m=+3170.596518745" watchObservedRunningTime="2026-10-31 04:00:17.631016086 +0000 UTC m=+3170.597349479"
Oct 31 04:00:17.668633 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.668606    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"
Oct 31 04:00:17.715807 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.715719    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8993f9fc-9cf4-4437-986c-1d43928ebd8b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8993f9fc-9cf4-4437-986c-1d43928ebd8b-maas-api-service-ca\") pod \"8993f9fc-9cf4-4437-986c-1d43928ebd8b\" (UID: \"8993f9fc-9cf4-4437-986c-1d43928ebd8b\") "
Oct 31 04:00:17.715951 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.715935    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8993f9fc-9cf4-4437-986c-1d43928ebd8b-kube-api-access-ldkc8\" (UniqueName: \"kubernetes.io/projected/8993f9fc-9cf4-4437-986c-1d43928ebd8b-kube-api-access-ldkc8\") pod \"8993f9fc-9cf4-4437-986c-1d43928ebd8b\" (UID: \"8993f9fc-9cf4-4437-986c-1d43928ebd8b\") "
Oct 31 04:00:17.716112 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.716091    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8993f9fc-9cf4-4437-986c-1d43928ebd8b-maas-api-service-ca" pod "8993f9fc-9cf4-4437-986c-1d43928ebd8b" (UID: "8993f9fc-9cf4-4437-986c-1d43928ebd8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:17.716307 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.716292    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8993f9fc-9cf4-4437-986c-1d43928ebd8b-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:17.717859 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.717835    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8993f9fc-9cf4-4437-986c-1d43928ebd8b-kube-api-access-ldkc8" pod "8993f9fc-9cf4-4437-986c-1d43928ebd8b" (UID: "8993f9fc-9cf4-4437-986c-1d43928ebd8b"). InnerVolumeSpecName "kube-api-access-ldkc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:17.729452 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.729430    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"]
Oct 31 04:00:17.817771 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.817741    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldkc8\" (UniqueName: \"kubernetes.io/projected/8993f9fc-9cf4-4437-986c-1d43928ebd8b-kube-api-access-ldkc8\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:17.856144 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.856074    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"]
Oct 31 04:00:17.860622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.860599    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890320-9z86g"]
Oct 31 04:00:17.860887 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.860872    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"
Oct 31 04:00:17.918559 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.918533    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1242b4c4-226f-4826-8900-656c58049ade-kube-api-access-vg4d4\" (UniqueName: \"kubernetes.io/projected/1242b4c4-226f-4826-8900-656c58049ade-kube-api-access-vg4d4\") pod \"1242b4c4-226f-4826-8900-656c58049ade\" (UID: \"1242b4c4-226f-4826-8900-656c58049ade\") "
Oct 31 04:00:17.918684 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.918602    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1242b4c4-226f-4826-8900-656c58049ade-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1242b4c4-226f-4826-8900-656c58049ade-maas-api-service-ca\") pod \"1242b4c4-226f-4826-8900-656c58049ade\" (UID: \"1242b4c4-226f-4826-8900-656c58049ade\") "
Oct 31 04:00:17.918996 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.918973    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1242b4c4-226f-4826-8900-656c58049ade-maas-api-service-ca" pod "1242b4c4-226f-4826-8900-656c58049ade" (UID: "1242b4c4-226f-4826-8900-656c58049ade"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:17.919260 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.919243    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1242b4c4-226f-4826-8900-656c58049ade-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:17.920629 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.920610    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1242b4c4-226f-4826-8900-656c58049ade-kube-api-access-vg4d4" pod "1242b4c4-226f-4826-8900-656c58049ade" (UID: "1242b4c4-226f-4826-8900-656c58049ade"). InnerVolumeSpecName "kube-api-access-vg4d4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:17.958097 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:17.958073    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"]
Oct 31 04:00:18.020749 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.020718    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg4d4\" (UniqueName: \"kubernetes.io/projected/1242b4c4-226f-4826-8900-656c58049ade-kube-api-access-vg4d4\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:18.090187 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.090141    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq" podStartSLOduration=18.090125046 podStartE2EDuration="18.090125046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:18.087604073 +0000 UTC m=+3171.053937417" watchObservedRunningTime="2026-10-31 04:00:18.090125046 +0000 UTC m=+3171.056458390"
Oct 31 04:00:18.531837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.531807    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"
Oct 31 04:00:18.531837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.531825    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5" event={"ID":"8993f9fc-9cf4-4437-986c-1d43928ebd8b","Type":"ContainerDied","Data":"cb1a38df560147fdff115c50757a828e23ab0ae32a79aa987fa3ca6f1a4b21e7"}
Oct 31 04:00:18.532342 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.531883    4900 scope.go:122] "RemoveContainer" containerID="9e0ebb06091f7e36d3cff2d38c2d11846781a2c71b686f164a5733979a59f09e"
Oct 31 04:00:18.533869 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.533844    4900 generic.go:356] "Generic (PLEG): container finished" podID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" containerID="d01e425a4c9ae0be13d5e883d5bceaf01fb8f28ff2712d8f8cfcca3d8f295c1c" exitCode=60
Oct 31 04:00:18.533994 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.533927    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq" event={"ID":"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085","Type":"ContainerDied","Data":"d01e425a4c9ae0be13d5e883d5bceaf01fb8f28ff2712d8f8cfcca3d8f295c1c"}
Oct 31 04:00:18.534332 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.534312    4900 scope.go:122] "RemoveContainer" containerID="d01e425a4c9ae0be13d5e883d5bceaf01fb8f28ff2712d8f8cfcca3d8f295c1c"
Oct 31 04:00:18.534614 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:18.534586    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq_odh-ai-gateway-infra(1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq" podUID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085"
Oct 31 04:00:18.537160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.537141    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"
Oct 31 04:00:18.537515 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.537143    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd" event={"ID":"1242b4c4-226f-4826-8900-656c58049ade","Type":"ContainerDied","Data":"625694e3a3cdce07722e07774101317425799204b5df3773ccb65c7bece490dd"}
Oct 31 04:00:18.539257 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.539236    4900 generic.go:356] "Generic (PLEG): container finished" podID="01b15a45-31af-4080-8597-edf79c2d3d8d" containerID="06fb92bff20900c3e9e618be53d9c86e0e598d31244407b2ea342d1fbada8c6d" exitCode=60
Oct 31 04:00:18.539373 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.539290    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v" event={"ID":"01b15a45-31af-4080-8597-edf79c2d3d8d","Type":"ContainerDied","Data":"06fb92bff20900c3e9e618be53d9c86e0e598d31244407b2ea342d1fbada8c6d"}
Oct 31 04:00:18.539562 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.539548    4900 scope.go:122] "RemoveContainer" containerID="06fb92bff20900c3e9e618be53d9c86e0e598d31244407b2ea342d1fbada8c6d"
Oct 31 04:00:18.539751 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:18.539733    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v_odh-ai-gateway-infra(01b15a45-31af-4080-8597-edf79c2d3d8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v" podUID="01b15a45-31af-4080-8597-edf79c2d3d8d"
Oct 31 04:00:18.542531 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.542516    4900 scope.go:122] "RemoveContainer" containerID="75f135734d1696f538f37942d813b8cc6be09744422ac9f7c1035b5c7ccdc7fc"
Oct 31 04:00:18.555241 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.555222    4900 scope.go:122] "RemoveContainer" containerID="47384cc9c035bf67b0e0f5c9695a91b08809355aff1f354c4ffddc29694b9205"
Oct 31 04:00:18.565347 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.565329    4900 scope.go:122] "RemoveContainer" containerID="b073d3290238887cfb7ac4f12207620d4abf02b200964f126738411557382b89"
Oct 31 04:00:18.641751 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.641725    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"]
Oct 31 04:00:18.682810 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.682769    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"
Oct 31 04:00:18.730190 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.730148    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad1d8dfc-1271-4776-9af8-6429d8a201bf-kube-api-access-dbksx\" (UniqueName: \"kubernetes.io/projected/ad1d8dfc-1271-4776-9af8-6429d8a201bf-kube-api-access-dbksx\") pod \"ad1d8dfc-1271-4776-9af8-6429d8a201bf\" (UID: \"ad1d8dfc-1271-4776-9af8-6429d8a201bf\") "
Oct 31 04:00:18.730319 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.730219    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad1d8dfc-1271-4776-9af8-6429d8a201bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1d8dfc-1271-4776-9af8-6429d8a201bf-maas-api-service-ca\") pod \"ad1d8dfc-1271-4776-9af8-6429d8a201bf\" (UID: \"ad1d8dfc-1271-4776-9af8-6429d8a201bf\") "
Oct 31 04:00:18.730609 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.730579    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad1d8dfc-1271-4776-9af8-6429d8a201bf-maas-api-service-ca" pod "ad1d8dfc-1271-4776-9af8-6429d8a201bf" (UID: "ad1d8dfc-1271-4776-9af8-6429d8a201bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:18.730924 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.730907    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad1d8dfc-1271-4776-9af8-6429d8a201bf-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:18.732204 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.732183    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1d8dfc-1271-4776-9af8-6429d8a201bf-kube-api-access-dbksx" pod "ad1d8dfc-1271-4776-9af8-6429d8a201bf" (UID: "ad1d8dfc-1271-4776-9af8-6429d8a201bf"). InnerVolumeSpecName "kube-api-access-dbksx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:18.832050 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.831975    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbksx\" (UniqueName: \"kubernetes.io/projected/ad1d8dfc-1271-4776-9af8-6429d8a201bf-kube-api-access-dbksx\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:18.925018 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.924975    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"]
Oct 31 04:00:18.927426 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:18.927398    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890320-rrwjd"]
Oct 31 04:00:19.037185 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.037158    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"]
Oct 31 04:00:19.041014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.040991    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-8b85a6-29890320-98ds5"]
Oct 31 04:00:19.100200 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.100138    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"]
Oct 31 04:00:19.154251 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.154219    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1242b4c4-226f-4826-8900-656c58049ade" path="/var/lib/kubelet/pods/1242b4c4-226f-4826-8900-656c58049ade/volumes"
Oct 31 04:00:19.154844 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.154825    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fcca104-5bd0-4498-b230-893bdcd090d1" path="/var/lib/kubelet/pods/5fcca104-5bd0-4498-b230-893bdcd090d1/volumes"
Oct 31 04:00:19.155362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.155345    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8993f9fc-9cf4-4437-986c-1d43928ebd8b" path="/var/lib/kubelet/pods/8993f9fc-9cf4-4437-986c-1d43928ebd8b/volumes"
Oct 31 04:00:19.550486 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.550465    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"
Oct 31 04:00:19.550897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.550465    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t" event={"ID":"ad1d8dfc-1271-4776-9af8-6429d8a201bf","Type":"ContainerDied","Data":"f39c6ce0686e87032aa257104501c8bac07757795313afc511de3f91e5c4ead3"}
Oct 31 04:00:19.550897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.550609    4900 scope.go:122] "RemoveContainer" containerID="4163e60052c7df3c241c5a62c8398d5eddb3f51a8bf5f9897394c5f18181d2c9"
Oct 31 04:00:19.696987 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.696959    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"
Oct 31 04:00:19.736254 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.736231    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"
Oct 31 04:00:19.742409 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.742388    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b15a45-31af-4080-8597-edf79c2d3d8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b15a45-31af-4080-8597-edf79c2d3d8d-maas-api-service-ca\") pod \"01b15a45-31af-4080-8597-edf79c2d3d8d\" (UID: \"01b15a45-31af-4080-8597-edf79c2d3d8d\") "
Oct 31 04:00:19.742487 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.742471    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b15a45-31af-4080-8597-edf79c2d3d8d-kube-api-access-hkhdb\" (UniqueName: \"kubernetes.io/projected/01b15a45-31af-4080-8597-edf79c2d3d8d-kube-api-access-hkhdb\") pod \"01b15a45-31af-4080-8597-edf79c2d3d8d\" (UID: \"01b15a45-31af-4080-8597-edf79c2d3d8d\") "
Oct 31 04:00:19.742733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.742711    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b15a45-31af-4080-8597-edf79c2d3d8d-maas-api-service-ca" pod "01b15a45-31af-4080-8597-edf79c2d3d8d" (UID: "01b15a45-31af-4080-8597-edf79c2d3d8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:19.742937 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.742921    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01b15a45-31af-4080-8597-edf79c2d3d8d-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.744476 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.744453    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b15a45-31af-4080-8597-edf79c2d3d8d-kube-api-access-hkhdb" pod "01b15a45-31af-4080-8597-edf79c2d3d8d" (UID: "01b15a45-31af-4080-8597-edf79c2d3d8d"). InnerVolumeSpecName "kube-api-access-hkhdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:19.773233 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.773202    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"]
Oct 31 04:00:19.776156 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.776134    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-ab9b62-29890320-8497t"]
Oct 31 04:00:19.843672 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.843580    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085-kube-api-access-gwmz5\" (UniqueName: \"kubernetes.io/projected/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085-kube-api-access-gwmz5\") pod \"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085\" (UID: \"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085\") "
Oct 31 04:00:19.843861 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.843754    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085-maas-api-service-ca\") pod \"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085\" (UID: \"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085\") "
Oct 31 04:00:19.844096 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.844073    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085-maas-api-service-ca" pod "1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" (UID: "1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:19.844295 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.844280    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkhdb\" (UniqueName: \"kubernetes.io/projected/01b15a45-31af-4080-8597-edf79c2d3d8d-kube-api-access-hkhdb\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.844345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.844300    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.845631 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.845610    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085-kube-api-access-gwmz5" pod "1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" (UID: "1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085"). InnerVolumeSpecName "kube-api-access-gwmz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:19.945799 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:19.945766    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwmz5\" (UniqueName: \"kubernetes.io/projected/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085-kube-api-access-gwmz5\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:20.559837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:20.559808    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"
Oct 31 04:00:20.560284 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:20.559839    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v" event={"ID":"01b15a45-31af-4080-8597-edf79c2d3d8d","Type":"ContainerDied","Data":"7fd41ef2caa93d55c1866ba902bdb64501a0639206b61d6a8a357b22a499108a"}
Oct 31 04:00:20.560284 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:20.559889    4900 scope.go:122] "RemoveContainer" containerID="06fb92bff20900c3e9e618be53d9c86e0e598d31244407b2ea342d1fbada8c6d"
Oct 31 04:00:20.561163 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:20.561145    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq" event={"ID":"1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085","Type":"ContainerDied","Data":"b695d119fb88a4ea903b925f5e1a669311c97c6cccfbff8042789e46033d975e"}
Oct 31 04:00:20.561232 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:20.561176    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"
Oct 31 04:00:20.571562 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:20.571540    4900 scope.go:122] "RemoveContainer" containerID="d01e425a4c9ae0be13d5e883d5bceaf01fb8f28ff2712d8f8cfcca3d8f295c1c"
Oct 31 04:00:20.724132 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:20.724100    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"]
Oct 31 04:00:20.774121 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:20.774090    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c0d50c-29890320-nxn4v"]
Oct 31 04:00:20.840228 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:20.840101    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"]
Oct 31 04:00:20.842374 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:20.842355    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890320-cwghq"]
Oct 31 04:00:21.160550 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:21.160467    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b15a45-31af-4080-8597-edf79c2d3d8d" path="/var/lib/kubelet/pods/01b15a45-31af-4080-8597-edf79c2d3d8d/volumes"
Oct 31 04:00:21.161000 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:21.160978    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" path="/var/lib/kubelet/pods/1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085/volumes"
Oct 31 04:00:21.161411 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:21.161394    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad1d8dfc-1271-4776-9af8-6429d8a201bf" path="/var/lib/kubelet/pods/ad1d8dfc-1271-4776-9af8-6429d8a201bf/volumes"
Oct 31 04:00:22.210937 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:22.210899    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:27.111401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:27.111366    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:32.005529 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:32.005494    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:36.884227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:36.884195    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:36.984290 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:00:36.984256    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:00:41.836073 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:41.836040    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:46.619168 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:46.619133    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:51.067418 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:51.067384    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:51.587436 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:51.587405    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:53.372976 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.372942    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"]
Oct 31 04:00:53.373487 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373473    4900 cpu_manager.go:414] "containerMap: removing container" podUID="1242b4c4-226f-4826-8900-656c58049ade" containerName="cleanup"
Oct 31 04:00:53.373536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373488    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1242b4c4-226f-4826-8900-656c58049ade" containerName="cleanup"
Oct 31 04:00:53.373536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373498    4900 cpu_manager.go:414] "containerMap: removing container" podUID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" containerName="cleanup"
Oct 31 04:00:53.373536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373504    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" containerName="cleanup"
Oct 31 04:00:53.373536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373516    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ad1d8dfc-1271-4776-9af8-6429d8a201bf" containerName="cleanup"
Oct 31 04:00:53.373536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373521    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1d8dfc-1271-4776-9af8-6429d8a201bf" containerName="cleanup"
Oct 31 04:00:53.373536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373529    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ad1d8dfc-1271-4776-9af8-6429d8a201bf" containerName="cleanup"
Oct 31 04:00:53.373536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373534    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1d8dfc-1271-4776-9af8-6429d8a201bf" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373551    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ad1d8dfc-1271-4776-9af8-6429d8a201bf" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373555    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1d8dfc-1271-4776-9af8-6429d8a201bf" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373564    4900 cpu_manager.go:414] "containerMap: removing container" podUID="5fcca104-5bd0-4498-b230-893bdcd090d1" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373569    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcca104-5bd0-4498-b230-893bdcd090d1" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373575    4900 cpu_manager.go:414] "containerMap: removing container" podUID="5fcca104-5bd0-4498-b230-893bdcd090d1" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373580    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcca104-5bd0-4498-b230-893bdcd090d1" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373586    4900 cpu_manager.go:414] "containerMap: removing container" podUID="01b15a45-31af-4080-8597-edf79c2d3d8d" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373590    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b15a45-31af-4080-8597-edf79c2d3d8d" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373597    4900 cpu_manager.go:414] "containerMap: removing container" podUID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373602    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373612    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8993f9fc-9cf4-4437-986c-1d43928ebd8b" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373615    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8993f9fc-9cf4-4437-986c-1d43928ebd8b" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373621    4900 cpu_manager.go:414] "containerMap: removing container" podUID="01b15a45-31af-4080-8597-edf79c2d3d8d" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373625    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b15a45-31af-4080-8597-edf79c2d3d8d" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373631    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8993f9fc-9cf4-4437-986c-1d43928ebd8b" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373636    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8993f9fc-9cf4-4437-986c-1d43928ebd8b" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373644    4900 cpu_manager.go:414] "containerMap: removing container" podUID="5fcca104-5bd0-4498-b230-893bdcd090d1" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373650    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcca104-5bd0-4498-b230-893bdcd090d1" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373661    4900 cpu_manager.go:414] "containerMap: removing container" podUID="1242b4c4-226f-4826-8900-656c58049ade" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373665    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1242b4c4-226f-4826-8900-656c58049ade" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373739    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="1242b4c4-226f-4826-8900-656c58049ade" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373748    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8993f9fc-9cf4-4437-986c-1d43928ebd8b" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373755    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b15a45-31af-4080-8597-edf79c2d3d8d" containerName="cleanup"
Oct 31 04:00:53.373752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373761    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8993f9fc-9cf4-4437-986c-1d43928ebd8b" containerName="cleanup"
Oct 31 04:00:53.374495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373768    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcca104-5bd0-4498-b230-893bdcd090d1" containerName="cleanup"
Oct 31 04:00:53.374495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373775    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8993f9fc-9cf4-4437-986c-1d43928ebd8b" containerName="cleanup"
Oct 31 04:00:53.374495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373782    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="1242b4c4-226f-4826-8900-656c58049ade" containerName="cleanup"
Oct 31 04:00:53.374495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373800    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b15a45-31af-4080-8597-edf79c2d3d8d" containerName="cleanup"
Oct 31 04:00:53.374495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373808    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="1242b4c4-226f-4826-8900-656c58049ade" containerName="cleanup"
Oct 31 04:00:53.374495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373814    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" containerName="cleanup"
Oct 31 04:00:53.374495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373820    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1d8dfc-1271-4776-9af8-6429d8a201bf" containerName="cleanup"
Oct 31 04:00:53.374495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373826    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1d8dfc-1271-4776-9af8-6429d8a201bf" containerName="cleanup"
Oct 31 04:00:53.374495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373833    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1d8dfc-1271-4776-9af8-6429d8a201bf" containerName="cleanup"
Oct 31 04:00:53.374495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373838    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcca104-5bd0-4498-b230-893bdcd090d1" containerName="cleanup"
Oct 31 04:00:53.374495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.373844    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" containerName="cleanup"
Oct 31 04:00:53.377042 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.377026    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"
Oct 31 04:00:53.430382 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.430347    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"]
Oct 31 04:00:53.449314 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.449275    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmtrh\" (UniqueName: \"kubernetes.io/projected/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-kube-api-access-cmtrh\") pod \"maas-api-revoke-sub-61784e4f-b7zv8\" (UID: \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"
Oct 31 04:00:53.449507 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.449336    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61784e4f-b7zv8\" (UID: \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"
Oct 31 04:00:53.449507 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.449428    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61784e4f-b7zv8\" (UID: \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"
Oct 31 04:00:53.476903 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.476872    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:00:53.477076 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.476922    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:00:53.550244 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.550204    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmtrh\" (UniqueName: \"kubernetes.io/projected/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-kube-api-access-cmtrh\") pod \"maas-api-revoke-sub-61784e4f-b7zv8\" (UID: \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"
Oct 31 04:00:53.550428 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.550264    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61784e4f-b7zv8\" (UID: \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"
Oct 31 04:00:53.550428 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.550314    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61784e4f-b7zv8\" (UID: \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"
Oct 31 04:00:53.551079 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.551052    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61784e4f-b7zv8\" (UID: \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"
Oct 31 04:00:53.552590 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.552564    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61784e4f-b7zv8\" (UID: \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"
Oct 31 04:00:53.607395 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.607367    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmtrh\" (UniqueName: \"kubernetes.io/projected/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-kube-api-access-cmtrh\") pod \"maas-api-revoke-sub-61784e4f-b7zv8\" (UID: \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"
Oct 31 04:00:53.689689 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:53.689605    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"
Oct 31 04:00:54.066408 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:00:54.066371    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf79a1c1_06f1_4f9d_bd60_de74abfaf581.slice/crio-345caa77513b4f65f5d5f33ce3f8b6af98f060a70c197867202143d2b73bc6e9 WatchSource:0}: Error finding container 345caa77513b4f65f5d5f33ce3f8b6af98f060a70c197867202143d2b73bc6e9: Status 404 returned error can't find the container with id 345caa77513b4f65f5d5f33ce3f8b6af98f060a70c197867202143d2b73bc6e9
Oct 31 04:00:54.068017 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:54.067991    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"]
Oct 31 04:00:54.761052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:54.761011    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8" event={"ID":"bf79a1c1-06f1-4f9d-bd60-de74abfaf581","Type":"ContainerStarted","Data":"b6bb1e1a4666c5b7e95b98231c3bb674e0a7df821bd441f34f41eea6482b3851"}
Oct 31 04:00:54.761513 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:54.761075    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8" event={"ID":"bf79a1c1-06f1-4f9d-bd60-de74abfaf581","Type":"ContainerStarted","Data":"345caa77513b4f65f5d5f33ce3f8b6af98f060a70c197867202143d2b73bc6e9"}
Oct 31 04:00:54.875826 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:54.875748    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8" podStartSLOduration=1.875731756 podStartE2EDuration="1.875731756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:54.873855747 +0000 UTC m=+3207.840189092" watchObservedRunningTime="2026-10-31 04:00:54.875731756 +0000 UTC m=+3207.842065131"
Oct 31 04:00:55.768210 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:55.768170    4900 generic.go:356] "Generic (PLEG): container finished" podID="bf79a1c1-06f1-4f9d-bd60-de74abfaf581" containerID="b6bb1e1a4666c5b7e95b98231c3bb674e0a7df821bd441f34f41eea6482b3851" exitCode=0
Oct 31 04:00:55.768687 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:55.768249    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8" event={"ID":"bf79a1c1-06f1-4f9d-bd60-de74abfaf581","Type":"ContainerDied","Data":"b6bb1e1a4666c5b7e95b98231c3bb674e0a7df821bd441f34f41eea6482b3851"}
Oct 31 04:00:56.910701 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:56.910672    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"
Oct 31 04:00:57.096258 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:57.096163    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-kube-api-access-cmtrh\" (UniqueName: \"kubernetes.io/projected/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-kube-api-access-cmtrh\") pod \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\" (UID: \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\") "
Oct 31 04:00:57.096258 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:57.096247    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-maas-api-xks-ca\") pod \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\" (UID: \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\") "
Oct 31 04:00:57.096511 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:57.096400    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-maas-api-openshift-ca\") pod \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\" (UID: \"bf79a1c1-06f1-4f9d-bd60-de74abfaf581\") "
Oct 31 04:00:57.096760 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:57.096735    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-maas-api-openshift-ca" pod "bf79a1c1-06f1-4f9d-bd60-de74abfaf581" (UID: "bf79a1c1-06f1-4f9d-bd60-de74abfaf581"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:57.098339 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:57.098308    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-kube-api-access-cmtrh" pod "bf79a1c1-06f1-4f9d-bd60-de74abfaf581" (UID: "bf79a1c1-06f1-4f9d-bd60-de74abfaf581"). InnerVolumeSpecName "kube-api-access-cmtrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:57.098438 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:57.098385    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-maas-api-xks-ca" pod "bf79a1c1-06f1-4f9d-bd60-de74abfaf581" (UID: "bf79a1c1-06f1-4f9d-bd60-de74abfaf581"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:00:57.198036 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:57.197996    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmtrh\" (UniqueName: \"kubernetes.io/projected/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-kube-api-access-cmtrh\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:57.198036 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:57.198031    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:57.198036 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:57.198042    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf79a1c1-06f1-4f9d-bd60-de74abfaf581-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:57.204243 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:57.204213    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:57.425027 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:57.424948    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:00:57.781068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:57.781038    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"
Oct 31 04:00:57.781068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:57.781048    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8" event={"ID":"bf79a1c1-06f1-4f9d-bd60-de74abfaf581","Type":"ContainerDied","Data":"345caa77513b4f65f5d5f33ce3f8b6af98f060a70c197867202143d2b73bc6e9"}
Oct 31 04:00:57.781324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:00:57.781097    4900 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="345caa77513b4f65f5d5f33ce3f8b6af98f060a70c197867202143d2b73bc6e9"
Oct 31 04:01:00.288820 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:00.288759    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:00.529031 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:00.528998    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:01.406169 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:01.406120    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:03.763188 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:03.763154    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:04.148759 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.148671    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"]
Oct 31 04:01:04.149246 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.149231    4900 cpu_manager.go:414] "containerMap: removing container" podUID="01b15a45-31af-4080-8597-edf79c2d3d8d" containerName="cleanup"
Oct 31 04:01:04.149294 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.149247    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b15a45-31af-4080-8597-edf79c2d3d8d" containerName="cleanup"
Oct 31 04:01:04.149294 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.149257    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8993f9fc-9cf4-4437-986c-1d43928ebd8b" containerName="cleanup"
Oct 31 04:01:04.149294 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.149262    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8993f9fc-9cf4-4437-986c-1d43928ebd8b" containerName="cleanup"
Oct 31 04:01:04.149294 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.149284    4900 cpu_manager.go:414] "containerMap: removing container" podUID="1242b4c4-226f-4826-8900-656c58049ade" containerName="cleanup"
Oct 31 04:01:04.149294 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.149289    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1242b4c4-226f-4826-8900-656c58049ade" containerName="cleanup"
Oct 31 04:01:04.149294 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.149295    4900 cpu_manager.go:414] "containerMap: removing container" podUID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" containerName="cleanup"
Oct 31 04:01:04.149479 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.149300    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" containerName="cleanup"
Oct 31 04:01:04.149479 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.149313    4900 cpu_manager.go:414] "containerMap: removing container" podUID="bf79a1c1-06f1-4f9d-bd60-de74abfaf581" containerName="revoke-keys"
Oct 31 04:01:04.149479 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.149318    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf79a1c1-06f1-4f9d-bd60-de74abfaf581" containerName="revoke-keys"
Oct 31 04:01:04.149479 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.149396    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcca104-5bd0-4498-b230-893bdcd090d1" containerName="cleanup"
Oct 31 04:01:04.149479 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.149406    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b15a45-31af-4080-8597-edf79c2d3d8d" containerName="cleanup"
Oct 31 04:01:04.149479 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.149414    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef4fed3-5ba5-4af9-b3fb-9f3ae8b57085" containerName="cleanup"
Oct 31 04:01:04.149479 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.149421    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf79a1c1-06f1-4f9d-bd60-de74abfaf581" containerName="revoke-keys"
Oct 31 04:01:04.152716 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.152701    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"
Oct 31 04:01:04.184361 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.184325    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fdd6806-dz5mg\" (UID: \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"
Oct 31 04:01:04.184520 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.184463    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fdd6806-dz5mg\" (UID: \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"
Oct 31 04:01:04.184520 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.184512    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b7sp\" (UniqueName: \"kubernetes.io/projected/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-kube-api-access-6b7sp\") pod \"maas-api-revoke-sub-5fdd6806-dz5mg\" (UID: \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"
Oct 31 04:01:04.205569 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.205534    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"]
Oct 31 04:01:04.252178 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.252144    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:01:04.252178 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.252156    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:01:04.285688 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.285653    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fdd6806-dz5mg\" (UID: \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"
Oct 31 04:01:04.285896 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.285710    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fdd6806-dz5mg\" (UID: \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"
Oct 31 04:01:04.285896 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.285734    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b7sp\" (UniqueName: \"kubernetes.io/projected/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-kube-api-access-6b7sp\") pod \"maas-api-revoke-sub-5fdd6806-dz5mg\" (UID: \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"
Oct 31 04:01:04.286314 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.286293    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fdd6806-dz5mg\" (UID: \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"
Oct 31 04:01:04.288134 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.288110    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fdd6806-dz5mg\" (UID: \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"
Oct 31 04:01:04.343270 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.343237    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b7sp\" (UniqueName: \"kubernetes.io/projected/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-kube-api-access-6b7sp\") pod \"maas-api-revoke-sub-5fdd6806-dz5mg\" (UID: \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"
Oct 31 04:01:04.475013 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.474928    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"
Oct 31 04:01:04.844026 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:01:04.843989    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb2b71f_e32a_4b82_93f6_b48e3b41ca35.slice/crio-d9c159faf6a4c98ae02f919d54697a0bff61037dc146026a2a14c471475d88af WatchSource:0}: Error finding container d9c159faf6a4c98ae02f919d54697a0bff61037dc146026a2a14c471475d88af: Status 404 returned error can't find the container with id d9c159faf6a4c98ae02f919d54697a0bff61037dc146026a2a14c471475d88af
Oct 31 04:01:04.846299 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:04.846275    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"]
Oct 31 04:01:05.829520 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:05.829482    4900 generic.go:356] "Generic (PLEG): container finished" podID="0cb2b71f-e32a-4b82-93f6-b48e3b41ca35" containerID="e6e581793d5f16dbccd29faa1103ac23b1d326e7e00a3def20754a24f805ae09" exitCode=0
Oct 31 04:01:05.829687 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:05.829566    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg" event={"ID":"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35","Type":"ContainerDied","Data":"e6e581793d5f16dbccd29faa1103ac23b1d326e7e00a3def20754a24f805ae09"}
Oct 31 04:01:05.829687 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:05.829648    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg" event={"ID":"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35","Type":"ContainerStarted","Data":"d9c159faf6a4c98ae02f919d54697a0bff61037dc146026a2a14c471475d88af"}
Oct 31 04:01:06.304722 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:06.304689    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:06.988514 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:06.988484    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"
Oct 31 04:01:06.989802 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:06.989762    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:01:07.122127 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:07.122035    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-maas-api-xks-ca\") pod \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\" (UID: \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\") "
Oct 31 04:01:07.122127 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:07.122081    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-maas-api-openshift-ca\") pod \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\" (UID: \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\") "
Oct 31 04:01:07.122368 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:07.122142    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-kube-api-access-6b7sp\" (UniqueName: \"kubernetes.io/projected/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-kube-api-access-6b7sp\") pod \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\" (UID: \"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35\") "
Oct 31 04:01:07.122566 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:07.122531    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-maas-api-openshift-ca" pod "0cb2b71f-e32a-4b82-93f6-b48e3b41ca35" (UID: "0cb2b71f-e32a-4b82-93f6-b48e3b41ca35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:07.122998 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:07.122974    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:07.124351 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:07.124328    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-maas-api-xks-ca" pod "0cb2b71f-e32a-4b82-93f6-b48e3b41ca35" (UID: "0cb2b71f-e32a-4b82-93f6-b48e3b41ca35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:01:07.124463 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:07.124440    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-kube-api-access-6b7sp" pod "0cb2b71f-e32a-4b82-93f6-b48e3b41ca35" (UID: "0cb2b71f-e32a-4b82-93f6-b48e3b41ca35"). InnerVolumeSpecName "kube-api-access-6b7sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:07.224386 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:07.224349    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:07.224386 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:07.224386    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b7sp\" (UniqueName: \"kubernetes.io/projected/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35-kube-api-access-6b7sp\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:07.842604 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:07.842574    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"
Oct 31 04:01:07.843115 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:07.842574    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg" event={"ID":"0cb2b71f-e32a-4b82-93f6-b48e3b41ca35","Type":"ContainerDied","Data":"d9c159faf6a4c98ae02f919d54697a0bff61037dc146026a2a14c471475d88af"}
Oct 31 04:01:07.843115 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:07.842703    4900 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9c159faf6a4c98ae02f919d54697a0bff61037dc146026a2a14c471475d88af"
Oct 31 04:01:10.841961 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:10.841928    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"]
Oct 31 04:01:10.842511 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:10.842498    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0cb2b71f-e32a-4b82-93f6-b48e3b41ca35" containerName="revoke-keys"
Oct 31 04:01:10.842558 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:10.842512    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb2b71f-e32a-4b82-93f6-b48e3b41ca35" containerName="revoke-keys"
Oct 31 04:01:10.842614 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:10.842605    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb2b71f-e32a-4b82-93f6-b48e3b41ca35" containerName="revoke-keys"
Oct 31 04:01:10.845864 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:10.845850    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:10.898477 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:10.898450    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-856ddc-serving-cert\""
Oct 31 04:01:10.951283 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:10.951244    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"]
Oct 31 04:01:10.969148 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:10.969117    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f00e6e-3938-47ae-acc4-9a37950c7056-openshift-service-ca\") pod \"maas-api-e2e-explicit-856ddc-6865b96887-srp9c\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:10.969324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:10.969169    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-maas-api-tls\") pod \"maas-api-e2e-explicit-856ddc-6865b96887-srp9c\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:10.969324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:10.969277    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-metrics-certs\") pod \"maas-api-e2e-explicit-856ddc-6865b96887-srp9c\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:10.969493 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:10.969470    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lggxs\" (UniqueName: \"kubernetes.io/projected/e1f00e6e-3938-47ae-acc4-9a37950c7056-kube-api-access-lggxs\") pod \"maas-api-e2e-explicit-856ddc-6865b96887-srp9c\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:11.070449 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:11.070418    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lggxs\" (UniqueName: \"kubernetes.io/projected/e1f00e6e-3938-47ae-acc4-9a37950c7056-kube-api-access-lggxs\") pod \"maas-api-e2e-explicit-856ddc-6865b96887-srp9c\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:11.070618 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:11.070501    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f00e6e-3938-47ae-acc4-9a37950c7056-openshift-service-ca\") pod \"maas-api-e2e-explicit-856ddc-6865b96887-srp9c\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:11.070618 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:11.070533    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-maas-api-tls\") pod \"maas-api-e2e-explicit-856ddc-6865b96887-srp9c\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:11.070618 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:11.070576    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-metrics-certs\") pod \"maas-api-e2e-explicit-856ddc-6865b96887-srp9c\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:11.070727 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:11.070698    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-serving-cert: secret "maas-api-e2e-explicit-856ddc-serving-cert" not found
Oct 31 04:01:11.070797 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:11.070775    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-maas-api-tls podName:e1f00e6e-3938-47ae-acc4-9a37950c7056 nodeName:}" failed. No retries permitted until 2026-10-31 04:01:11.570758924 +0000 UTC m=+3224.537092253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-maas-api-tls") pod "maas-api-e2e-explicit-856ddc-6865b96887-srp9c" (UID: "e1f00e6e-3938-47ae-acc4-9a37950c7056") : secret "maas-api-e2e-explicit-856ddc-serving-cert" not found
Oct 31 04:01:11.071255 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:11.071230    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f00e6e-3938-47ae-acc4-9a37950c7056-openshift-service-ca\") pod \"maas-api-e2e-explicit-856ddc-6865b96887-srp9c\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:11.072942 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:11.072922    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-metrics-certs\") pod \"maas-api-e2e-explicit-856ddc-6865b96887-srp9c\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:11.129772 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:11.129701    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lggxs\" (UniqueName: \"kubernetes.io/projected/e1f00e6e-3938-47ae-acc4-9a37950c7056-kube-api-access-lggxs\") pod \"maas-api-e2e-explicit-856ddc-6865b96887-srp9c\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:11.133360 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:11.133331    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:11.359972 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:11.359940    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:11.576702 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:11.576671    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-maas-api-tls\") pod \"maas-api-e2e-explicit-856ddc-6865b96887-srp9c\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:11.579029 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:11.579010    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-maas-api-tls\") pod \"maas-api-e2e-explicit-856ddc-6865b96887-srp9c\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:11.584732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:11.584707    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:11.759229 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:11.759193    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:12.126181 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:01:12.126121    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f00e6e_3938_47ae_acc4_9a37950c7056.slice/crio-0506f03e47732df87f27e786740bb43e060b45d5053e0e20049e18c7e9f34c61 WatchSource:0}: Error finding container 0506f03e47732df87f27e786740bb43e060b45d5053e0e20049e18c7e9f34c61: Status 404 returned error can't find the container with id 0506f03e47732df87f27e786740bb43e060b45d5053e0e20049e18c7e9f34c61
Oct 31 04:01:12.127953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:12.127931    4900 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:01:12.128823 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:12.128781    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"]
Oct 31 04:01:12.875409 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:12.875377    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c" event={"ID":"e1f00e6e-3938-47ae-acc4-9a37950c7056","Type":"ContainerStarted","Data":"0506f03e47732df87f27e786740bb43e060b45d5053e0e20049e18c7e9f34c61"}
Oct 31 04:01:13.882570 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:13.882537    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c" event={"ID":"e1f00e6e-3938-47ae-acc4-9a37950c7056","Type":"ContainerStarted","Data":"eb36f405b2c3805140e0cc3b4e41571236a7461c6c8ae190287ddb6a1bd2023d"}
Oct 31 04:01:13.882989 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:13.882670    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:14.003027 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:14.002973    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c" podStartSLOduration=2.995739558 podStartE2EDuration="4.002959551s" totalImagesPullingTime="1.007219993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:14.000076799 +0000 UTC m=+3226.966410164" watchObservedRunningTime="2026-10-31 04:01:14.002959551 +0000 UTC m=+3226.969292922"
Oct 31 04:01:16.317717 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:16.317681    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:18.187039 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:18.186995    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:19.624430 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:19.624385    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:19.750322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:19.750286    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7205-2b0ef4"]
Oct 31 04:01:19.754318 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:19.754302    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-2b0ef4"
Oct 31 04:01:19.805127 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:19.805094    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7205-2b0ef4"]
Oct 31 04:01:19.852115 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:19.852090    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c4jjr\""
Oct 31 04:01:19.876458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:19.876377    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jxcd\" (UniqueName: \"kubernetes.io/projected/048b0184-863d-4fac-8f9f-b4ea19752438-kube-api-access-8jxcd\") pod \"test-curl-7205-2b0ef4\" (UID: \"048b0184-863d-4fac-8f9f-b4ea19752438\") " pod="opendatahub/test-curl-7205-2b0ef4"
Oct 31 04:01:19.893873 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:19.893845    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:01:19.977564 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:19.977530    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jxcd\" (UniqueName: \"kubernetes.io/projected/048b0184-863d-4fac-8f9f-b4ea19752438-kube-api-access-8jxcd\") pod \"test-curl-7205-2b0ef4\" (UID: \"048b0184-863d-4fac-8f9f-b4ea19752438\") " pod="opendatahub/test-curl-7205-2b0ef4"
Oct 31 04:01:20.034238 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:20.034205    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jxcd\" (UniqueName: \"kubernetes.io/projected/048b0184-863d-4fac-8f9f-b4ea19752438-kube-api-access-8jxcd\") pod \"test-curl-7205-2b0ef4\" (UID: \"048b0184-863d-4fac-8f9f-b4ea19752438\") " pod="opendatahub/test-curl-7205-2b0ef4"
Oct 31 04:01:20.066334 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:20.066302    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-2b0ef4"
Oct 31 04:01:20.452078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:20.452053    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7205-2b0ef4"]
Oct 31 04:01:20.923347 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:20.923309    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7205-2b0ef4" event={"ID":"048b0184-863d-4fac-8f9f-b4ea19752438","Type":"ContainerStarted","Data":"886c438ea8851ac3b19b79bd25a2a818d1e12896ab13d9aebd6c1c11f4792e18"}
Oct 31 04:01:21.201725 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:21.201645    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:23.947311 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:23.947273    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7205-2b0ef4" event={"ID":"048b0184-863d-4fac-8f9f-b4ea19752438","Type":"ContainerStarted","Data":"46118048beb946ef37453f04bb34bb36da470b82494ce6b9ac2e306d731f8247"}
Oct 31 04:01:24.060014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:24.059959    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7205-2b0ef4" podStartSLOduration=2.568949253 podStartE2EDuration="5.059940798s" totalImagesPullingTime="2.490991545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:24.059359781 +0000 UTC m=+3237.025693124" watchObservedRunningTime="2026-10-31 04:01:24.059940798 +0000 UTC m=+3237.026274146"
Oct 31 04:01:25.443948 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:25.443915    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7205-2b0ef4"]
Oct 31 04:01:25.446770 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:25.446742    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7205-2b0ef4"]
Oct 31 04:01:25.606202 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:25.606168    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7205-70ec18"]
Oct 31 04:01:25.606716 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:25.606703    4900 cpu_manager.go:414] "containerMap: removing container" podUID="048b0184-863d-4fac-8f9f-b4ea19752438" containerName="test-curl-7205-2b0ef4"
Oct 31 04:01:25.606765 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:25.606717    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048b0184-863d-4fac-8f9f-b4ea19752438" containerName="test-curl-7205-2b0ef4"
Oct 31 04:01:25.606825 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:25.606820    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="048b0184-863d-4fac-8f9f-b4ea19752438" containerName="test-curl-7205-2b0ef4"
Oct 31 04:01:25.610243 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:25.610228    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-70ec18"
Oct 31 04:01:25.647735 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:25.647702    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnj4w\" (UniqueName: \"kubernetes.io/projected/3a1bf8a6-101e-4814-a194-62c84f820ea2-kube-api-access-mnj4w\") pod \"test-curl-7205-70ec18\" (UID: \"3a1bf8a6-101e-4814-a194-62c84f820ea2\") " pod="opendatahub/test-curl-7205-70ec18"
Oct 31 04:01:25.660817 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:25.660780    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7205-70ec18"]
Oct 31 04:01:25.749548 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:25.749512    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnj4w\" (UniqueName: \"kubernetes.io/projected/3a1bf8a6-101e-4814-a194-62c84f820ea2-kube-api-access-mnj4w\") pod \"test-curl-7205-70ec18\" (UID: \"3a1bf8a6-101e-4814-a194-62c84f820ea2\") " pod="opendatahub/test-curl-7205-70ec18"
Oct 31 04:01:25.806813 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:25.806775    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnj4w\" (UniqueName: \"kubernetes.io/projected/3a1bf8a6-101e-4814-a194-62c84f820ea2-kube-api-access-mnj4w\") pod \"test-curl-7205-70ec18\" (UID: \"3a1bf8a6-101e-4814-a194-62c84f820ea2\") " pod="opendatahub/test-curl-7205-70ec18"
Oct 31 04:01:25.921900 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:25.921855    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-70ec18"
Oct 31 04:01:25.960637 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:25.960589    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7205-2b0ef4" podUID="048b0184-863d-4fac-8f9f-b4ea19752438" containerName="test-curl-7205-2b0ef4" containerID="cri-o://46118048beb946ef37453f04bb34bb36da470b82494ce6b9ac2e306d731f8247" gracePeriod=2
Oct 31 04:01:26.012259 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:26.012183    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-2b0ef4\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="048b0184-863d-4fac-8f9f-b4ea19752438" pod="opendatahub/test-curl-7205-2b0ef4"
Oct 31 04:01:26.077938 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:26.077910    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:26.289699 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:01:26.289670    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a1bf8a6_101e_4814_a194_62c84f820ea2.slice/crio-b426382c44dc1006d33ac7d9900f8e514e29edcf6454dddec164c933a115d740 WatchSource:0}: Error finding container b426382c44dc1006d33ac7d9900f8e514e29edcf6454dddec164c933a115d740: Status 404 returned error can't find the container with id b426382c44dc1006d33ac7d9900f8e514e29edcf6454dddec164c933a115d740
Oct 31 04:01:26.292243 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:26.292223    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7205-70ec18"]
Oct 31 04:01:26.968502 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:26.968465    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7205-70ec18" event={"ID":"3a1bf8a6-101e-4814-a194-62c84f820ea2","Type":"ContainerStarted","Data":"f9ae1006ec16013379ba20a97f17a292cc99301bc0d2650802f895d06fa35b08"}
Oct 31 04:01:26.968502 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:26.968508    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7205-70ec18" event={"ID":"3a1bf8a6-101e-4814-a194-62c84f820ea2","Type":"ContainerStarted","Data":"b426382c44dc1006d33ac7d9900f8e514e29edcf6454dddec164c933a115d740"}
Oct 31 04:01:27.019877 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:27.019834    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-2b0ef4\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="048b0184-863d-4fac-8f9f-b4ea19752438" pod="opendatahub/test-curl-7205-2b0ef4"
Oct 31 04:01:27.132752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:27.132699    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7205-70ec18" podStartSLOduration=2.132685404 podStartE2EDuration="2.132685404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:27.130757637 +0000 UTC m=+3240.097090982" watchObservedRunningTime="2026-10-31 04:01:27.132685404 +0000 UTC m=+3240.099018747"
Oct 31 04:01:27.208208 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:27.208155    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-2b0ef4\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="048b0184-863d-4fac-8f9f-b4ea19752438" pod="opendatahub/test-curl-7205-2b0ef4"
Oct 31 04:01:28.187466 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.187441    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-2b0ef4"
Oct 31 04:01:28.279971 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.279929    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048b0184-863d-4fac-8f9f-b4ea19752438-kube-api-access-8jxcd\" (UniqueName: \"kubernetes.io/projected/048b0184-863d-4fac-8f9f-b4ea19752438-kube-api-access-8jxcd\") pod \"048b0184-863d-4fac-8f9f-b4ea19752438\" (UID: \"048b0184-863d-4fac-8f9f-b4ea19752438\") "
Oct 31 04:01:28.281949 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.281918    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048b0184-863d-4fac-8f9f-b4ea19752438-kube-api-access-8jxcd" pod "048b0184-863d-4fac-8f9f-b4ea19752438" (UID: "048b0184-863d-4fac-8f9f-b4ea19752438"). InnerVolumeSpecName "kube-api-access-8jxcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:28.382002 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.381965    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jxcd\" (UniqueName: \"kubernetes.io/projected/048b0184-863d-4fac-8f9f-b4ea19752438-kube-api-access-8jxcd\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:28.601434 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.601341    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7205-70ec18"]
Oct 31 04:01:28.604278 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.604247    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7205-70ec18"]
Oct 31 04:01:28.971971 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.971881    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7205-4dcafd"]
Oct 31 04:01:28.972444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.972430    4900 cpu_manager.go:414] "containerMap: removing container" podUID="3a1bf8a6-101e-4814-a194-62c84f820ea2" containerName="test-curl-7205-70ec18"
Oct 31 04:01:28.972549 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.972445    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1bf8a6-101e-4814-a194-62c84f820ea2" containerName="test-curl-7205-70ec18"
Oct 31 04:01:28.972604 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.972560    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1bf8a6-101e-4814-a194-62c84f820ea2" containerName="test-curl-7205-70ec18"
Oct 31 04:01:28.975992 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.975972    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-4dcafd"
Oct 31 04:01:28.983464 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.983436    4900 generic.go:356] "Generic (PLEG): container finished" podID="048b0184-863d-4fac-8f9f-b4ea19752438" containerID="46118048beb946ef37453f04bb34bb36da470b82494ce6b9ac2e306d731f8247" exitCode=137
Oct 31 04:01:28.983657 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.983487    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-2b0ef4"
Oct 31 04:01:28.983657 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.983521    4900 scope.go:122] "RemoveContainer" containerID="46118048beb946ef37453f04bb34bb36da470b82494ce6b9ac2e306d731f8247"
Oct 31 04:01:28.984199 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.983961    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7205-70ec18" podUID="3a1bf8a6-101e-4814-a194-62c84f820ea2" containerName="test-curl-7205-70ec18" containerID="cri-o://f9ae1006ec16013379ba20a97f17a292cc99301bc0d2650802f895d06fa35b08" gracePeriod=2
Oct 31 04:01:28.994386 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.994364    4900 scope.go:122] "RemoveContainer" containerID="46118048beb946ef37453f04bb34bb36da470b82494ce6b9ac2e306d731f8247"
Oct 31 04:01:28.994654 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:28.994636    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46118048beb946ef37453f04bb34bb36da470b82494ce6b9ac2e306d731f8247\": container with ID starting with 46118048beb946ef37453f04bb34bb36da470b82494ce6b9ac2e306d731f8247 not found: ID does not exist" containerID="46118048beb946ef37453f04bb34bb36da470b82494ce6b9ac2e306d731f8247"
Oct 31 04:01:28.994697 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:28.994663    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46118048beb946ef37453f04bb34bb36da470b82494ce6b9ac2e306d731f8247"} err="failed to get container status \"46118048beb946ef37453f04bb34bb36da470b82494ce6b9ac2e306d731f8247\": rpc error: code = NotFound desc = could not find container \"46118048beb946ef37453f04bb34bb36da470b82494ce6b9ac2e306d731f8247\": container with ID starting with 46118048beb946ef37453f04bb34bb36da470b82494ce6b9ac2e306d731f8247 not found: ID does not exist"
Oct 31 04:01:29.024500 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:29.024474    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7205-4dcafd"]
Oct 31 04:01:29.090226 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:29.090196    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjm9x\" (UniqueName: \"kubernetes.io/projected/e87fc128-335e-45f7-b2d9-be141a4c94a9-kube-api-access-tjm9x\") pod \"test-curl-7205-4dcafd\" (UID: \"e87fc128-335e-45f7-b2d9-be141a4c94a9\") " pod="opendatahub/test-curl-7205-4dcafd"
Oct 31 04:01:29.137634 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:29.137598    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-2b0ef4\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="048b0184-863d-4fac-8f9f-b4ea19752438" pod="opendatahub/test-curl-7205-2b0ef4"
Oct 31 04:01:29.153435 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:29.153407    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048b0184-863d-4fac-8f9f-b4ea19752438" path="/var/lib/kubelet/pods/048b0184-863d-4fac-8f9f-b4ea19752438/volumes"
Oct 31 04:01:29.188857 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:29.188813    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-70ec18\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="3a1bf8a6-101e-4814-a194-62c84f820ea2" pod="opendatahub/test-curl-7205-70ec18"
Oct 31 04:01:29.191475 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:29.191447    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjm9x\" (UniqueName: \"kubernetes.io/projected/e87fc128-335e-45f7-b2d9-be141a4c94a9-kube-api-access-tjm9x\") pod \"test-curl-7205-4dcafd\" (UID: \"e87fc128-335e-45f7-b2d9-be141a4c94a9\") " pod="opendatahub/test-curl-7205-4dcafd"
Oct 31 04:01:29.248843 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:29.248813    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjm9x\" (UniqueName: \"kubernetes.io/projected/e87fc128-335e-45f7-b2d9-be141a4c94a9-kube-api-access-tjm9x\") pod \"test-curl-7205-4dcafd\" (UID: \"e87fc128-335e-45f7-b2d9-be141a4c94a9\") " pod="opendatahub/test-curl-7205-4dcafd"
Oct 31 04:01:29.290423 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:29.290393    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-4dcafd"
Oct 31 04:01:29.658003 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:01:29.657959    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode87fc128_335e_45f7_b2d9_be141a4c94a9.slice/crio-ae8b10f4918a2740ee264bd6ae3e1c7e4b957d1988ff35cf5cc2cf83856e9350 WatchSource:0}: Error finding container ae8b10f4918a2740ee264bd6ae3e1c7e4b957d1988ff35cf5cc2cf83856e9350: Status 404 returned error can't find the container with id ae8b10f4918a2740ee264bd6ae3e1c7e4b957d1988ff35cf5cc2cf83856e9350
Oct 31 04:01:29.660027 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:29.660002    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7205-4dcafd"]
Oct 31 04:01:29.999639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:29.999599    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7205-4dcafd" event={"ID":"e87fc128-335e-45f7-b2d9-be141a4c94a9","Type":"ContainerStarted","Data":"f343f6a669ed99b5a59686e7f8afdc1eaf9bd45f798927a70bccef5d7890c36a"}
Oct 31 04:01:29.999639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:29.999644    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7205-4dcafd" event={"ID":"e87fc128-335e-45f7-b2d9-be141a4c94a9","Type":"ContainerStarted","Data":"ae8b10f4918a2740ee264bd6ae3e1c7e4b957d1988ff35cf5cc2cf83856e9350"}
Oct 31 04:01:30.115621 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:30.115551    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7205-4dcafd" podStartSLOduration=2.115531506 podStartE2EDuration="2.115531506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:30.111761767 +0000 UTC m=+3243.078095111" watchObservedRunningTime="2026-10-31 04:01:30.115531506 +0000 UTC m=+3243.081864852"
Oct 31 04:01:31.093779 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.093745    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:31.253008 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.252941    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-70ec18"
Oct 31 04:01:31.284145 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.284114    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"]
Oct 31 04:01:31.284368 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.284347    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr" podUID="bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" containerName="istio-proxy" containerID="cri-o://6e1eadb67b60968a148ceb76ed995fe3146a606d00dd0599eac18910e7a0d1c1" gracePeriod=30
Oct 31 04:01:31.320479 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.320448    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1bf8a6-101e-4814-a194-62c84f820ea2-kube-api-access-mnj4w\" (UniqueName: \"kubernetes.io/projected/3a1bf8a6-101e-4814-a194-62c84f820ea2-kube-api-access-mnj4w\") pod \"3a1bf8a6-101e-4814-a194-62c84f820ea2\" (UID: \"3a1bf8a6-101e-4814-a194-62c84f820ea2\") "
Oct 31 04:01:31.323340 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.323285    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1bf8a6-101e-4814-a194-62c84f820ea2-kube-api-access-mnj4w" pod "3a1bf8a6-101e-4814-a194-62c84f820ea2" (UID: "3a1bf8a6-101e-4814-a194-62c84f820ea2"). InnerVolumeSpecName "kube-api-access-mnj4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:31.338639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.338608    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"]
Oct 31 04:01:31.346960 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.346922    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p" podUID="6ee9cc43-db19-4ef8-9a17-c576d409d498" containerName="istio-proxy" containerID="cri-o://20b51f0779bca56dbf1dfca73b761d60b891c7df3541fadad043bfe8f2bb5fb6" gracePeriod=30
Oct 31 04:01:31.422803 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.422756    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnj4w\" (UniqueName: \"kubernetes.io/projected/3a1bf8a6-101e-4814-a194-62c84f820ea2-kube-api-access-mnj4w\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:31.513219 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.512370    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7205-4dcafd"]
Oct 31 04:01:31.516524 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.516444    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7205-4dcafd"]
Oct 31 04:01:31.659917 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.659886    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7205-7a9a33"]
Oct 31 04:01:31.660419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.660405    4900 cpu_manager.go:414] "containerMap: removing container" podUID="e87fc128-335e-45f7-b2d9-be141a4c94a9" containerName="test-curl-7205-4dcafd"
Oct 31 04:01:31.660465 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.660420    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e87fc128-335e-45f7-b2d9-be141a4c94a9" containerName="test-curl-7205-4dcafd"
Oct 31 04:01:31.660545 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.660536    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="e87fc128-335e-45f7-b2d9-be141a4c94a9" containerName="test-curl-7205-4dcafd"
Oct 31 04:01:31.663892 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.663875    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-7a9a33"
Oct 31 04:01:31.716828 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.713749    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7205-7a9a33"]
Oct 31 04:01:31.728066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.728033    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwfz6\" (UniqueName: \"kubernetes.io/projected/bf806ec6-ac0a-43d2-ba2f-920878f2dc99-kube-api-access-lwfz6\") pod \"test-curl-7205-7a9a33\" (UID: \"bf806ec6-ac0a-43d2-ba2f-920878f2dc99\") " pod="opendatahub/test-curl-7205-7a9a33"
Oct 31 04:01:31.829650 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.829566    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwfz6\" (UniqueName: \"kubernetes.io/projected/bf806ec6-ac0a-43d2-ba2f-920878f2dc99-kube-api-access-lwfz6\") pod \"test-curl-7205-7a9a33\" (UID: \"bf806ec6-ac0a-43d2-ba2f-920878f2dc99\") " pod="opendatahub/test-curl-7205-7a9a33"
Oct 31 04:01:31.886233 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.886196    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwfz6\" (UniqueName: \"kubernetes.io/projected/bf806ec6-ac0a-43d2-ba2f-920878f2dc99-kube-api-access-lwfz6\") pod \"test-curl-7205-7a9a33\" (UID: \"bf806ec6-ac0a-43d2-ba2f-920878f2dc99\") " pod="opendatahub/test-curl-7205-7a9a33"
Oct 31 04:01:31.979971 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:31.979932    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-7a9a33"
Oct 31 04:01:32.014192 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:32.014160    4900 generic.go:356] "Generic (PLEG): container finished" podID="3a1bf8a6-101e-4814-a194-62c84f820ea2" containerID="f9ae1006ec16013379ba20a97f17a292cc99301bc0d2650802f895d06fa35b08" exitCode=137
Oct 31 04:01:32.014363 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:32.014205    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-70ec18"
Oct 31 04:01:32.014363 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:32.014251    4900 scope.go:122] "RemoveContainer" containerID="f9ae1006ec16013379ba20a97f17a292cc99301bc0d2650802f895d06fa35b08"
Oct 31 04:01:32.014484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:32.014462    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7205-4dcafd" podUID="e87fc128-335e-45f7-b2d9-be141a4c94a9" containerName="test-curl-7205-4dcafd" containerID="cri-o://f343f6a669ed99b5a59686e7f8afdc1eaf9bd45f798927a70bccef5d7890c36a" gracePeriod=2
Oct 31 04:01:32.026017 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:32.025990    4900 scope.go:122] "RemoveContainer" containerID="f9ae1006ec16013379ba20a97f17a292cc99301bc0d2650802f895d06fa35b08"
Oct 31 04:01:32.026399 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:32.026371    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ae1006ec16013379ba20a97f17a292cc99301bc0d2650802f895d06fa35b08\": container with ID starting with f9ae1006ec16013379ba20a97f17a292cc99301bc0d2650802f895d06fa35b08 not found: ID does not exist" containerID="f9ae1006ec16013379ba20a97f17a292cc99301bc0d2650802f895d06fa35b08"
Oct 31 04:01:32.026505 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:32.026411    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ae1006ec16013379ba20a97f17a292cc99301bc0d2650802f895d06fa35b08"} err="failed to get container status \"f9ae1006ec16013379ba20a97f17a292cc99301bc0d2650802f895d06fa35b08\": rpc error: code = NotFound desc = could not find container \"f9ae1006ec16013379ba20a97f17a292cc99301bc0d2650802f895d06fa35b08\": container with ID starting with f9ae1006ec16013379ba20a97f17a292cc99301bc0d2650802f895d06fa35b08 not found: ID does not exist"
Oct 31 04:01:32.066441 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:32.066402    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-4dcafd\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="e87fc128-335e-45f7-b2d9-be141a4c94a9" pod="opendatahub/test-curl-7205-4dcafd"
Oct 31 04:01:32.117670 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:32.117583    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-70ec18\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="3a1bf8a6-101e-4814-a194-62c84f820ea2" pod="opendatahub/test-curl-7205-70ec18"
Oct 31 04:01:32.168548 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:32.168509    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-4dcafd\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="e87fc128-335e-45f7-b2d9-be141a4c94a9" pod="opendatahub/test-curl-7205-4dcafd"
Oct 31 04:01:32.349859 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:01:32.349827    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf806ec6_ac0a_43d2_ba2f_920878f2dc99.slice/crio-c042e55e2187e3003d5730a0e658e24542c1e7bde644ad2d248f7f55db601261 WatchSource:0}: Error finding container c042e55e2187e3003d5730a0e658e24542c1e7bde644ad2d248f7f55db601261: Status 404 returned error can't find the container with id c042e55e2187e3003d5730a0e658e24542c1e7bde644ad2d248f7f55db601261
Oct 31 04:01:32.353142 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:32.353119    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7205-7a9a33"]
Oct 31 04:01:33.022679 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:33.022646    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7205-7a9a33" event={"ID":"bf806ec6-ac0a-43d2-ba2f-920878f2dc99","Type":"ContainerStarted","Data":"5158af0f87d515928be073076c3d1dddd7b9370c4ed810d8c43be4eddf86806e"}
Oct 31 04:01:33.022940 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:33.022691    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7205-7a9a33" event={"ID":"bf806ec6-ac0a-43d2-ba2f-920878f2dc99","Type":"ContainerStarted","Data":"c042e55e2187e3003d5730a0e658e24542c1e7bde644ad2d248f7f55db601261"}
Oct 31 04:01:33.073913 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:33.073869    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-70ec18\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="3a1bf8a6-101e-4814-a194-62c84f820ea2" pod="opendatahub/test-curl-7205-70ec18"
Oct 31 04:01:33.124870 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:33.124836    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-4dcafd\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="e87fc128-335e-45f7-b2d9-be141a4c94a9" pod="opendatahub/test-curl-7205-4dcafd"
Oct 31 04:01:33.168006 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:33.167973    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1bf8a6-101e-4814-a194-62c84f820ea2" path="/var/lib/kubelet/pods/3a1bf8a6-101e-4814-a194-62c84f820ea2/volumes"
Oct 31 04:01:33.765760 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:33.765724    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:34.005193 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.005154    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:01:34.005471 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.005220    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p" podUID="6ee9cc43-db19-4ef8-9a17-c576d409d498" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:01:34.159429 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.159385    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:34.295675 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.295649    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-4dcafd"
Oct 31 04:01:34.380259 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.380223    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e87fc128-335e-45f7-b2d9-be141a4c94a9-kube-api-access-tjm9x\" (UniqueName: \"kubernetes.io/projected/e87fc128-335e-45f7-b2d9-be141a4c94a9-kube-api-access-tjm9x\") pod \"e87fc128-335e-45f7-b2d9-be141a4c94a9\" (UID: \"e87fc128-335e-45f7-b2d9-be141a4c94a9\") "
Oct 31 04:01:34.382360 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.382327    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e87fc128-335e-45f7-b2d9-be141a4c94a9-kube-api-access-tjm9x" pod "e87fc128-335e-45f7-b2d9-be141a4c94a9" (UID: "e87fc128-335e-45f7-b2d9-be141a4c94a9"). InnerVolumeSpecName "kube-api-access-tjm9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:34.481843 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.481807    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjm9x\" (UniqueName: \"kubernetes.io/projected/e87fc128-335e-45f7-b2d9-be141a4c94a9-kube-api-access-tjm9x\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:34.750812 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.741914    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7205-7a9a33" podStartSLOduration=3.741891622 podStartE2EDuration="3.741891622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:33.259591705 +0000 UTC m=+3246.225925050" watchObservedRunningTime="2026-10-31 04:01:34.741891622 +0000 UTC m=+3247.708224968"
Oct 31 04:01:34.770822 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.769866    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7205-7a9a33"]
Oct 31 04:01:34.774915 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.774767    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7205-7a9a33"]
Oct 31 04:01:34.880261 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.880215    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:34.911223 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.911182    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7205-671367"]
Oct 31 04:01:34.911932 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.911911    4900 cpu_manager.go:414] "containerMap: removing container" podUID="bf806ec6-ac0a-43d2-ba2f-920878f2dc99" containerName="test-curl-7205-7a9a33"
Oct 31 04:01:34.911932 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.911930    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf806ec6-ac0a-43d2-ba2f-920878f2dc99" containerName="test-curl-7205-7a9a33"
Oct 31 04:01:34.912129 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.912034    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf806ec6-ac0a-43d2-ba2f-920878f2dc99" containerName="test-curl-7205-7a9a33"
Oct 31 04:01:34.915404 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.915386    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-671367"
Oct 31 04:01:34.968050 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.968016    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7205-671367"]
Oct 31 04:01:34.988884 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:34.988310    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9jjh\" (UniqueName: \"kubernetes.io/projected/3af0d029-dc09-4f77-beb6-ac44c2dc2560-kube-api-access-b9jjh\") pod \"test-curl-7205-671367\" (UID: \"3af0d029-dc09-4f77-beb6-ac44c2dc2560\") " pod="opendatahub/test-curl-7205-671367"
Oct 31 04:01:35.063486 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:35.062322    4900 generic.go:356] "Generic (PLEG): container finished" podID="e87fc128-335e-45f7-b2d9-be141a4c94a9" containerID="f343f6a669ed99b5a59686e7f8afdc1eaf9bd45f798927a70bccef5d7890c36a" exitCode=137
Oct 31 04:01:35.063486 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:35.062518    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7205-7a9a33" podUID="bf806ec6-ac0a-43d2-ba2f-920878f2dc99" containerName="test-curl-7205-7a9a33" containerID="cri-o://5158af0f87d515928be073076c3d1dddd7b9370c4ed810d8c43be4eddf86806e" gracePeriod=2
Oct 31 04:01:35.063486 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:35.062763    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-4dcafd"
Oct 31 04:01:35.063486 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:35.062923    4900 scope.go:122] "RemoveContainer" containerID="f343f6a669ed99b5a59686e7f8afdc1eaf9bd45f798927a70bccef5d7890c36a"
Oct 31 04:01:35.076021 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:35.076001    4900 scope.go:122] "RemoveContainer" containerID="f343f6a669ed99b5a59686e7f8afdc1eaf9bd45f798927a70bccef5d7890c36a"
Oct 31 04:01:35.076843 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:35.076819    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f343f6a669ed99b5a59686e7f8afdc1eaf9bd45f798927a70bccef5d7890c36a\": container with ID starting with f343f6a669ed99b5a59686e7f8afdc1eaf9bd45f798927a70bccef5d7890c36a not found: ID does not exist" containerID="f343f6a669ed99b5a59686e7f8afdc1eaf9bd45f798927a70bccef5d7890c36a"
Oct 31 04:01:35.076920 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:35.076854    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f343f6a669ed99b5a59686e7f8afdc1eaf9bd45f798927a70bccef5d7890c36a"} err="failed to get container status \"f343f6a669ed99b5a59686e7f8afdc1eaf9bd45f798927a70bccef5d7890c36a\": rpc error: code = NotFound desc = could not find container \"f343f6a669ed99b5a59686e7f8afdc1eaf9bd45f798927a70bccef5d7890c36a\": container with ID starting with f343f6a669ed99b5a59686e7f8afdc1eaf9bd45f798927a70bccef5d7890c36a not found: ID does not exist"
Oct 31 04:01:35.089768 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:35.089743    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9jjh\" (UniqueName: \"kubernetes.io/projected/3af0d029-dc09-4f77-beb6-ac44c2dc2560-kube-api-access-b9jjh\") pod \"test-curl-7205-671367\" (UID: \"3af0d029-dc09-4f77-beb6-ac44c2dc2560\") " pod="opendatahub/test-curl-7205-671367"
Oct 31 04:01:35.105737 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:35.105706    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:35.113948 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:35.113918    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-7a9a33\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="bf806ec6-ac0a-43d2-ba2f-920878f2dc99" pod="opendatahub/test-curl-7205-7a9a33"
Oct 31 04:01:35.161888 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:35.161340    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9jjh\" (UniqueName: \"kubernetes.io/projected/3af0d029-dc09-4f77-beb6-ac44c2dc2560-kube-api-access-b9jjh\") pod \"test-curl-7205-671367\" (UID: \"3af0d029-dc09-4f77-beb6-ac44c2dc2560\") " pod="opendatahub/test-curl-7205-671367"
Oct 31 04:01:35.166962 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:35.166927    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-4dcafd\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="e87fc128-335e-45f7-b2d9-be141a4c94a9" pod="opendatahub/test-curl-7205-4dcafd"
Oct 31 04:01:35.173135 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:35.173090    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e87fc128-335e-45f7-b2d9-be141a4c94a9" path="/var/lib/kubelet/pods/e87fc128-335e-45f7-b2d9-be141a4c94a9/volumes"
Oct 31 04:01:35.217915 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:35.217867    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-7a9a33\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="bf806ec6-ac0a-43d2-ba2f-920878f2dc99" pod="opendatahub/test-curl-7205-7a9a33"
Oct 31 04:01:35.228549 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:35.228519    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-671367"
Oct 31 04:01:35.626063 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:01:35.626028    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3af0d029_dc09_4f77_beb6_ac44c2dc2560.slice/crio-2142de003bdcba8301b536a5130f3bc33d0bd2526d7e2aabf0caf18edd28145f WatchSource:0}: Error finding container 2142de003bdcba8301b536a5130f3bc33d0bd2526d7e2aabf0caf18edd28145f: Status 404 returned error can't find the container with id 2142de003bdcba8301b536a5130f3bc33d0bd2526d7e2aabf0caf18edd28145f
Oct 31 04:01:35.628133 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:35.628108    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7205-671367"]
Oct 31 04:01:36.076291 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.076251    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7205-671367" event={"ID":"3af0d029-dc09-4f77-beb6-ac44c2dc2560","Type":"ContainerStarted","Data":"f0972700a4b0e99906e8034847a5e0ea34a8e19a0c044483475307e5fd6aaed2"}
Oct 31 04:01:36.076503 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.076303    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7205-671367" event={"ID":"3af0d029-dc09-4f77-beb6-ac44c2dc2560","Type":"ContainerStarted","Data":"2142de003bdcba8301b536a5130f3bc33d0bd2526d7e2aabf0caf18edd28145f"}
Oct 31 04:01:36.190431 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.190367    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7205-671367" podStartSLOduration=2.190350658 podStartE2EDuration="2.190350658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:36.188419768 +0000 UTC m=+3249.154753112" watchObservedRunningTime="2026-10-31 04:01:36.190350658 +0000 UTC m=+3249.156684007"
Oct 31 04:01:36.334562 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:36.334479    4900 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda1fa9b_5f39_44cb_9f1d_8d64876b7dbe.slice/crio-conmon-6e1eadb67b60968a148ceb76ed995fe3146a606d00dd0599eac18910e7a0d1c1.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 04:01:36.405522 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.405497    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:36.472389 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.472359    4900 patch_prober.go:29] interesting pod/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Oct 31 04:01:36.472554 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.472413    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr" podUID="bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Oct 31 04:01:36.612180 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.612112    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 04:01:36.705411 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.705389    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 04:01:36.709066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.709046    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-token\" (UniqueName: \"kubernetes.io/projected/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-token\") pod \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") "
Oct 31 04:01:36.709169 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.709117    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-workload-socket\") pod \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") "
Oct 31 04:01:36.709169 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.709155    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-data\") pod \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") "
Oct 31 04:01:36.709245 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.709232    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istiod-ca-cert\") pod \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") "
Oct 31 04:01:36.709287 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.709250    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-envoy\") pod \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") "
Oct 31 04:01:36.709321 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.709287    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-workload-certs\") pod \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") "
Oct 31 04:01:36.709357 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.709332    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-credential-socket\") pod \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") "
Oct 31 04:01:36.709389 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.709352    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-kube-api-access-kvvj2\" (UniqueName: \"kubernetes.io/projected/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-kube-api-access-kvvj2\") pod \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") "
Oct 31 04:01:36.709462 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.709432    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-podinfo\") pod \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\" (UID: \"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe\") "
Oct 31 04:01:36.709649 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.709614    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-workload-socket" pod "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" (UID: "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:36.709719 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.709655    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-workload-certs" pod "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" (UID: "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:36.709719 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.709691    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-credential-socket" pod "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" (UID: "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:36.709719 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.709707    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istiod-ca-cert" pod "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" (UID: "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:36.709981 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.709961    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-data" pod "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" (UID: "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:36.710167 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.710144    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.710167 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.710165    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.710288 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.710175    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.710288 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.710184    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.710288 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.710192    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.711585 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.711563    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-token" pod "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" (UID: "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:36.712202 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.712180    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-kube-api-access-kvvj2" pod "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" (UID: "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe"). InnerVolumeSpecName "kube-api-access-kvvj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:36.712509 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.712477    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-podinfo" pod "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" (UID: "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:01:36.712655 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.712636    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-envoy" pod "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" (UID: "bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:36.811149 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.811113    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-credential-socket\") pod \"6ee9cc43-db19-4ef8-9a17-c576d409d498\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") "
Oct 31 04:01:36.811337 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.811220    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-data\") pod \"6ee9cc43-db19-4ef8-9a17-c576d409d498\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") "
Oct 31 04:01:36.811337 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.811296    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee9cc43-db19-4ef8-9a17-c576d409d498-kube-api-access-qkbl9\" (UniqueName: \"kubernetes.io/projected/6ee9cc43-db19-4ef8-9a17-c576d409d498-kube-api-access-qkbl9\") pod \"6ee9cc43-db19-4ef8-9a17-c576d409d498\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") "
Oct 31 04:01:36.811337 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.811327    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-podinfo\") pod \"6ee9cc43-db19-4ef8-9a17-c576d409d498\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") "
Oct 31 04:01:36.811512 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.811365    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-credential-socket" pod "6ee9cc43-db19-4ef8-9a17-c576d409d498" (UID: "6ee9cc43-db19-4ef8-9a17-c576d409d498"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:36.811512 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.811435    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-token\" (UniqueName: \"kubernetes.io/projected/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-token\") pod \"6ee9cc43-db19-4ef8-9a17-c576d409d498\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") "
Oct 31 04:01:36.811512 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.811489    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-workload-socket\") pod \"6ee9cc43-db19-4ef8-9a17-c576d409d498\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") "
Oct 31 04:01:36.811512 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.811497    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-data" pod "6ee9cc43-db19-4ef8-9a17-c576d409d498" (UID: "6ee9cc43-db19-4ef8-9a17-c576d409d498"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:36.811774 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.811533    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-workload-certs\") pod \"6ee9cc43-db19-4ef8-9a17-c576d409d498\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") "
Oct 31 04:01:36.811774 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.811567    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-envoy\") pod \"6ee9cc43-db19-4ef8-9a17-c576d409d498\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") "
Oct 31 04:01:36.811774 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.811613    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee9cc43-db19-4ef8-9a17-c576d409d498-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee9cc43-db19-4ef8-9a17-c576d409d498-istiod-ca-cert\") pod \"6ee9cc43-db19-4ef8-9a17-c576d409d498\" (UID: \"6ee9cc43-db19-4ef8-9a17-c576d409d498\") "
Oct 31 04:01:36.811774 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.811678    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-workload-socket" pod "6ee9cc43-db19-4ef8-9a17-c576d409d498" (UID: "6ee9cc43-db19-4ef8-9a17-c576d409d498"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:36.811995 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.811778    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-workload-certs" pod "6ee9cc43-db19-4ef8-9a17-c576d409d498" (UID: "6ee9cc43-db19-4ef8-9a17-c576d409d498"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:36.812147 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.812122    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee9cc43-db19-4ef8-9a17-c576d409d498-istiod-ca-cert" pod "6ee9cc43-db19-4ef8-9a17-c576d409d498" (UID: "6ee9cc43-db19-4ef8-9a17-c576d409d498"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:01:36.812262 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.812243    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvvj2\" (UniqueName: \"kubernetes.io/projected/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-kube-api-access-kvvj2\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.812322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.812270    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.812322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.812288    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.812322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.812302    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.812322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.812317    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.812548 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.812332    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee9cc43-db19-4ef8-9a17-c576d409d498-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.812548 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.812345    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.812548 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.812358    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.812548 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.812372    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.813574 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.813544    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-podinfo" pod "6ee9cc43-db19-4ef8-9a17-c576d409d498" (UID: "6ee9cc43-db19-4ef8-9a17-c576d409d498"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:01:36.813706 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.813577    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee9cc43-db19-4ef8-9a17-c576d409d498-kube-api-access-qkbl9" pod "6ee9cc43-db19-4ef8-9a17-c576d409d498" (UID: "6ee9cc43-db19-4ef8-9a17-c576d409d498"). InnerVolumeSpecName "kube-api-access-qkbl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:36.813761 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.813731    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-token" pod "6ee9cc43-db19-4ef8-9a17-c576d409d498" (UID: "6ee9cc43-db19-4ef8-9a17-c576d409d498"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:36.813848 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.813821    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-envoy" pod "6ee9cc43-db19-4ef8-9a17-c576d409d498" (UID: "6ee9cc43-db19-4ef8-9a17-c576d409d498"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:01:36.913545 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.913470    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.913545 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.913499    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.913545 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.913510    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkbl9\" (UniqueName: \"kubernetes.io/projected/6ee9cc43-db19-4ef8-9a17-c576d409d498-kube-api-access-qkbl9\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.913545 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:36.913519    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee9cc43-db19-4ef8-9a17-c576d409d498-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:36.984975 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:36.984937    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:01:37.084364 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.084335    4900 generic.go:356] "Generic (PLEG): container finished" podID="6ee9cc43-db19-4ef8-9a17-c576d409d498" containerID="20b51f0779bca56dbf1dfca73b761d60b891c7df3541fadad043bfe8f2bb5fb6" exitCode=0
Oct 31 04:01:37.084502 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.084379    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"
Oct 31 04:01:37.084502 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.084416    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p" event={"ID":"6ee9cc43-db19-4ef8-9a17-c576d409d498","Type":"ContainerDied","Data":"20b51f0779bca56dbf1dfca73b761d60b891c7df3541fadad043bfe8f2bb5fb6"}
Oct 31 04:01:37.084502 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.084487    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p" event={"ID":"6ee9cc43-db19-4ef8-9a17-c576d409d498","Type":"ContainerDied","Data":"b5e053150794e20f15bac7602d4a83e25c5817cf58e40d3eb0f4a4ef7e3109ab"}
Oct 31 04:01:37.084640 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.084511    4900 scope.go:122] "RemoveContainer" containerID="20b51f0779bca56dbf1dfca73b761d60b891c7df3541fadad043bfe8f2bb5fb6"
Oct 31 04:01:37.086088 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.086065    4900 generic.go:356] "Generic (PLEG): container finished" podID="bf806ec6-ac0a-43d2-ba2f-920878f2dc99" containerID="5158af0f87d515928be073076c3d1dddd7b9370c4ed810d8c43be4eddf86806e" exitCode=137
Oct 31 04:01:37.087596 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.087577    4900 generic.go:356] "Generic (PLEG): container finished" podID="bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" containerID="6e1eadb67b60968a148ceb76ed995fe3146a606d00dd0599eac18910e7a0d1c1" exitCode=0
Oct 31 04:01:37.087689 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.087614    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"
Oct 31 04:01:37.087689 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.087663    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr" event={"ID":"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe","Type":"ContainerDied","Data":"6e1eadb67b60968a148ceb76ed995fe3146a606d00dd0599eac18910e7a0d1c1"}
Oct 31 04:01:37.087776 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.087714    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr" event={"ID":"bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe","Type":"ContainerDied","Data":"c7fdf1d9faecf1b680b48bcd2e98090ee237ef204dbcd1d4bcc9719ba9865305"}
Oct 31 04:01:37.127174 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.127156    4900 scope.go:122] "RemoveContainer" containerID="20b51f0779bca56dbf1dfca73b761d60b891c7df3541fadad043bfe8f2bb5fb6"
Oct 31 04:01:37.127493 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:37.127474    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b51f0779bca56dbf1dfca73b761d60b891c7df3541fadad043bfe8f2bb5fb6\": container with ID starting with 20b51f0779bca56dbf1dfca73b761d60b891c7df3541fadad043bfe8f2bb5fb6 not found: ID does not exist" containerID="20b51f0779bca56dbf1dfca73b761d60b891c7df3541fadad043bfe8f2bb5fb6"
Oct 31 04:01:37.127537 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.127507    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b51f0779bca56dbf1dfca73b761d60b891c7df3541fadad043bfe8f2bb5fb6"} err="failed to get container status \"20b51f0779bca56dbf1dfca73b761d60b891c7df3541fadad043bfe8f2bb5fb6\": rpc error: code = NotFound desc = could not find container \"20b51f0779bca56dbf1dfca73b761d60b891c7df3541fadad043bfe8f2bb5fb6\": container with ID starting with 20b51f0779bca56dbf1dfca73b761d60b891c7df3541fadad043bfe8f2bb5fb6 not found: ID does not exist"
Oct 31 04:01:37.127537 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.127524    4900 scope.go:122] "RemoveContainer" containerID="6e1eadb67b60968a148ceb76ed995fe3146a606d00dd0599eac18910e7a0d1c1"
Oct 31 04:01:37.163478 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.163454    4900 scope.go:122] "RemoveContainer" containerID="6e1eadb67b60968a148ceb76ed995fe3146a606d00dd0599eac18910e7a0d1c1"
Oct 31 04:01:37.164576 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:37.164353    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e1eadb67b60968a148ceb76ed995fe3146a606d00dd0599eac18910e7a0d1c1\": container with ID starting with 6e1eadb67b60968a148ceb76ed995fe3146a606d00dd0599eac18910e7a0d1c1 not found: ID does not exist" containerID="6e1eadb67b60968a148ceb76ed995fe3146a606d00dd0599eac18910e7a0d1c1"
Oct 31 04:01:37.164576 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.164391    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e1eadb67b60968a148ceb76ed995fe3146a606d00dd0599eac18910e7a0d1c1"} err="failed to get container status \"6e1eadb67b60968a148ceb76ed995fe3146a606d00dd0599eac18910e7a0d1c1\": rpc error: code = NotFound desc = could not find container \"6e1eadb67b60968a148ceb76ed995fe3146a606d00dd0599eac18910e7a0d1c1\": container with ID starting with 6e1eadb67b60968a148ceb76ed995fe3146a606d00dd0599eac18910e7a0d1c1 not found: ID does not exist"
Oct 31 04:01:37.277272 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.277243    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"]
Oct 31 04:01:37.280099 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.280077    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-7a9a33"
Oct 31 04:01:37.284024 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.284000    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-zlr4p"]
Oct 31 04:01:37.415538 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.415453    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"]
Oct 31 04:01:37.419510 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.419478    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9360eb-openshift-default-6b47969f67-wkbbr"]
Oct 31 04:01:37.419880 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.419861    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf806ec6-ac0a-43d2-ba2f-920878f2dc99-kube-api-access-lwfz6\" (UniqueName: \"kubernetes.io/projected/bf806ec6-ac0a-43d2-ba2f-920878f2dc99-kube-api-access-lwfz6\") pod \"bf806ec6-ac0a-43d2-ba2f-920878f2dc99\" (UID: \"bf806ec6-ac0a-43d2-ba2f-920878f2dc99\") "
Oct 31 04:01:37.421988 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.421961    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf806ec6-ac0a-43d2-ba2f-920878f2dc99-kube-api-access-lwfz6" pod "bf806ec6-ac0a-43d2-ba2f-920878f2dc99" (UID: "bf806ec6-ac0a-43d2-ba2f-920878f2dc99"). InnerVolumeSpecName "kube-api-access-lwfz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:37.467755 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:37.467712    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-7a9a33\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="bf806ec6-ac0a-43d2-ba2f-920878f2dc99" pod="opendatahub/test-curl-7205-7a9a33"
Oct 31 04:01:37.521848 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.521811    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwfz6\" (UniqueName: \"kubernetes.io/projected/bf806ec6-ac0a-43d2-ba2f-920878f2dc99-kube-api-access-lwfz6\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:37.622186 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.622143    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7205-671367"]
Oct 31 04:01:37.629893 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.629863    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7205-671367"]
Oct 31 04:01:37.795594 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.795554    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7205-5c0940"]
Oct 31 04:01:37.796194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.796166    4900 cpu_manager.go:414] "containerMap: removing container" podUID="6ee9cc43-db19-4ef8-9a17-c576d409d498" containerName="istio-proxy"
Oct 31 04:01:37.796194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.796191    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee9cc43-db19-4ef8-9a17-c576d409d498" containerName="istio-proxy"
Oct 31 04:01:37.796356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.796204    4900 cpu_manager.go:414] "containerMap: removing container" podUID="bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" containerName="istio-proxy"
Oct 31 04:01:37.796356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.796211    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" containerName="istio-proxy"
Oct 31 04:01:37.796356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.796247    4900 cpu_manager.go:414] "containerMap: removing container" podUID="3af0d029-dc09-4f77-beb6-ac44c2dc2560" containerName="test-curl-7205-671367"
Oct 31 04:01:37.796356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.796253    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af0d029-dc09-4f77-beb6-ac44c2dc2560" containerName="test-curl-7205-671367"
Oct 31 04:01:37.796356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.796329    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af0d029-dc09-4f77-beb6-ac44c2dc2560" containerName="test-curl-7205-671367"
Oct 31 04:01:37.796356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.796338    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee9cc43-db19-4ef8-9a17-c576d409d498" containerName="istio-proxy"
Oct 31 04:01:37.796356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.796344    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" containerName="istio-proxy"
Oct 31 04:01:37.799091 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.799077    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-5c0940"
Oct 31 04:01:37.860838 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:37.857702    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-7a9a33\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="bf806ec6-ac0a-43d2-ba2f-920878f2dc99" pod="opendatahub/test-curl-7205-7a9a33"
Oct 31 04:01:37.862080 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.862053    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7205-5c0940"]
Oct 31 04:01:37.926499 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:37.926462    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4clp\" (UniqueName: \"kubernetes.io/projected/b5a7ea61-b607-4436-85e8-c6f6b85fe31f-kube-api-access-x4clp\") pod \"test-curl-7205-5c0940\" (UID: \"b5a7ea61-b607-4436-85e8-c6f6b85fe31f\") " pod="opendatahub/test-curl-7205-5c0940"
Oct 31 04:01:38.027575 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:38.027541    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4clp\" (UniqueName: \"kubernetes.io/projected/b5a7ea61-b607-4436-85e8-c6f6b85fe31f-kube-api-access-x4clp\") pod \"test-curl-7205-5c0940\" (UID: \"b5a7ea61-b607-4436-85e8-c6f6b85fe31f\") " pod="opendatahub/test-curl-7205-5c0940"
Oct 31 04:01:38.085309 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:38.085226    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4clp\" (UniqueName: \"kubernetes.io/projected/b5a7ea61-b607-4436-85e8-c6f6b85fe31f-kube-api-access-x4clp\") pod \"test-curl-7205-5c0940\" (UID: \"b5a7ea61-b607-4436-85e8-c6f6b85fe31f\") " pod="opendatahub/test-curl-7205-5c0940"
Oct 31 04:01:38.097207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:38.097180    4900 scope.go:122] "RemoveContainer" containerID="5158af0f87d515928be073076c3d1dddd7b9370c4ed810d8c43be4eddf86806e"
Oct 31 04:01:38.097400 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:38.097306    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-7a9a33"
Oct 31 04:01:38.102930 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:38.102897    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7205-671367" podUID="3af0d029-dc09-4f77-beb6-ac44c2dc2560" containerName="test-curl-7205-671367" containerID="cri-o://f0972700a4b0e99906e8034847a5e0ea34a8e19a0c044483475307e5fd6aaed2" gracePeriod=2
Oct 31 04:01:38.110190 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:38.110174    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-5c0940"
Oct 31 04:01:38.148892 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:38.148849    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-7a9a33\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="bf806ec6-ac0a-43d2-ba2f-920878f2dc99" pod="opendatahub/test-curl-7205-7a9a33"
Oct 31 04:01:38.200034 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:38.199994    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-671367\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="3af0d029-dc09-4f77-beb6-ac44c2dc2560" pod="opendatahub/test-curl-7205-671367"
Oct 31 04:01:38.251143 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:38.251096    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-7a9a33\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="bf806ec6-ac0a-43d2-ba2f-920878f2dc99" pod="opendatahub/test-curl-7205-7a9a33"
Oct 31 04:01:38.489865 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:01:38.489830    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5a7ea61_b607_4436_85e8_c6f6b85fe31f.slice/crio-969fa7195bdf0e840c333d8dd798e7231d4f78de13a1a58e2b7454e27a0fcbe6 WatchSource:0}: Error finding container 969fa7195bdf0e840c333d8dd798e7231d4f78de13a1a58e2b7454e27a0fcbe6: Status 404 returned error can't find the container with id 969fa7195bdf0e840c333d8dd798e7231d4f78de13a1a58e2b7454e27a0fcbe6
Oct 31 04:01:38.491529 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:38.491439    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7205-5c0940"]
Oct 31 04:01:39.111990 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:39.111898    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7205-5c0940" event={"ID":"b5a7ea61-b607-4436-85e8-c6f6b85fe31f","Type":"ContainerStarted","Data":"efd4cf2d6e5247171e94123c46d612f895127dec9119204f97cb878428fca26f"}
Oct 31 04:01:39.111990 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:39.111944    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7205-5c0940" event={"ID":"b5a7ea61-b607-4436-85e8-c6f6b85fe31f","Type":"ContainerStarted","Data":"969fa7195bdf0e840c333d8dd798e7231d4f78de13a1a58e2b7454e27a0fcbe6"}
Oct 31 04:01:39.154908 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:39.154875    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee9cc43-db19-4ef8-9a17-c576d409d498" path="/var/lib/kubelet/pods/6ee9cc43-db19-4ef8-9a17-c576d409d498/volumes"
Oct 31 04:01:39.155381 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:39.155357    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe" path="/var/lib/kubelet/pods/bda1fa9b-5f39-44cb-9f1d-8d64876b7dbe/volumes"
Oct 31 04:01:39.155682 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:39.155670    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf806ec6-ac0a-43d2-ba2f-920878f2dc99" path="/var/lib/kubelet/pods/bf806ec6-ac0a-43d2-ba2f-920878f2dc99/volumes"
Oct 31 04:01:39.226929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:39.226858    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7205-5c0940" podStartSLOduration=2.226838273 podStartE2EDuration="2.226838273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:39.223248559 +0000 UTC m=+3252.189581925" watchObservedRunningTime="2026-10-31 04:01:39.226838273 +0000 UTC m=+3252.193171622"
Oct 31 04:01:39.274147 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:39.274110    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-671367\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="3af0d029-dc09-4f77-beb6-ac44c2dc2560" pod="opendatahub/test-curl-7205-671367"
Oct 31 04:01:39.325003 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:39.324953    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-7a9a33\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="bf806ec6-ac0a-43d2-ba2f-920878f2dc99" pod="opendatahub/test-curl-7205-7a9a33"
Oct 31 04:01:39.612138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:39.612086    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:40.369922 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:40.369857    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-671367"
Oct 31 04:01:40.455849 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:40.455824    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af0d029-dc09-4f77-beb6-ac44c2dc2560-kube-api-access-b9jjh\" (UniqueName: \"kubernetes.io/projected/3af0d029-dc09-4f77-beb6-ac44c2dc2560-kube-api-access-b9jjh\") pod \"3af0d029-dc09-4f77-beb6-ac44c2dc2560\" (UID: \"3af0d029-dc09-4f77-beb6-ac44c2dc2560\") "
Oct 31 04:01:40.457744 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:40.457721    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af0d029-dc09-4f77-beb6-ac44c2dc2560-kube-api-access-b9jjh" pod "3af0d029-dc09-4f77-beb6-ac44c2dc2560" (UID: "3af0d029-dc09-4f77-beb6-ac44c2dc2560"). InnerVolumeSpecName "kube-api-access-b9jjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:40.557959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:40.557925    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9jjh\" (UniqueName: \"kubernetes.io/projected/3af0d029-dc09-4f77-beb6-ac44c2dc2560-kube-api-access-b9jjh\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:40.724483 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:40.724397    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7205-5c0940"]
Oct 31 04:01:40.726817 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:40.726756    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7205-5c0940"]
Oct 31 04:01:41.126528 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:41.126481    4900 generic.go:356] "Generic (PLEG): container finished" podID="3af0d029-dc09-4f77-beb6-ac44c2dc2560" containerID="f0972700a4b0e99906e8034847a5e0ea34a8e19a0c044483475307e5fd6aaed2" exitCode=137
Oct 31 04:01:41.126528 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:41.126518    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-671367"
Oct 31 04:01:41.126800 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:41.126580    4900 scope.go:122] "RemoveContainer" containerID="f0972700a4b0e99906e8034847a5e0ea34a8e19a0c044483475307e5fd6aaed2"
Oct 31 04:01:41.126800 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:41.126768    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7205-5c0940" podUID="b5a7ea61-b607-4436-85e8-c6f6b85fe31f" containerName="test-curl-7205-5c0940" containerID="cri-o://efd4cf2d6e5247171e94123c46d612f895127dec9119204f97cb878428fca26f" gracePeriod=2
Oct 31 04:01:41.137356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:41.137331    4900 scope.go:122] "RemoveContainer" containerID="f0972700a4b0e99906e8034847a5e0ea34a8e19a0c044483475307e5fd6aaed2"
Oct 31 04:01:41.137637 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:41.137617    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0972700a4b0e99906e8034847a5e0ea34a8e19a0c044483475307e5fd6aaed2\": container with ID starting with f0972700a4b0e99906e8034847a5e0ea34a8e19a0c044483475307e5fd6aaed2 not found: ID does not exist" containerID="f0972700a4b0e99906e8034847a5e0ea34a8e19a0c044483475307e5fd6aaed2"
Oct 31 04:01:41.137730 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:41.137641    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0972700a4b0e99906e8034847a5e0ea34a8e19a0c044483475307e5fd6aaed2"} err="failed to get container status \"f0972700a4b0e99906e8034847a5e0ea34a8e19a0c044483475307e5fd6aaed2\": rpc error: code = NotFound desc = could not find container \"f0972700a4b0e99906e8034847a5e0ea34a8e19a0c044483475307e5fd6aaed2\": container with ID starting with f0972700a4b0e99906e8034847a5e0ea34a8e19a0c044483475307e5fd6aaed2 not found: ID does not exist"
Oct 31 04:01:41.153852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:41.153823    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3af0d029-dc09-4f77-beb6-ac44c2dc2560" path="/var/lib/kubelet/pods/3af0d029-dc09-4f77-beb6-ac44c2dc2560/volumes"
Oct 31 04:01:41.177954 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:41.177920    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-5c0940\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="b5a7ea61-b607-4436-85e8-c6f6b85fe31f" pod="opendatahub/test-curl-7205-5c0940"
Oct 31 04:01:41.773565 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:41.773529    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:42.134400 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:42.134320    4900 generic.go:356] "Generic (PLEG): container finished" podID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerID="a2549ed5802638c525b9c572795964804d4acd67036fefe848f7920d34368a45" exitCode=0
Oct 31 04:01:42.134576 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:42.134395    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" event={"ID":"2323f996-abc1-4f51-9ed6-2e96cde226bc","Type":"ContainerDied","Data":"a2549ed5802638c525b9c572795964804d4acd67036fefe848f7920d34368a45"}
Oct 31 04:01:42.134887 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:42.134867    4900 scope.go:122] "RemoveContainer" containerID="a2549ed5802638c525b9c572795964804d4acd67036fefe848f7920d34368a45"
Oct 31 04:01:42.136622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:42.136593    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:01:42.136708 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:42.136691    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:01:42.136751 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:42.136737    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:01:42.308887 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:42.308855    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 04:01:42.481434 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:42.481400    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:42.955811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:42.942489    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"]
Oct 31 04:01:42.955811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:42.943383    4900 cpu_manager.go:414] "containerMap: removing container" podUID="b5a7ea61-b607-4436-85e8-c6f6b85fe31f" containerName="test-curl-7205-5c0940"
Oct 31 04:01:42.955811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:42.943404    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5a7ea61-b607-4436-85e8-c6f6b85fe31f" containerName="test-curl-7205-5c0940"
Oct 31 04:01:42.955811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:42.943529    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5a7ea61-b607-4436-85e8-c6f6b85fe31f" containerName="test-curl-7205-5c0940"
Oct 31 04:01:42.955811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:42.946848    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.002506 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.002476    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"]
Oct 31 04:01:43.053044 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.053007    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c7585c-openshift-default-dockercfg-jnxt8\""
Oct 31 04:01:43.085886 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.085855    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-envoy\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.086066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.085910    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-workload-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.086114 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.086084    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br6f2\" (UniqueName: \"kubernetes.io/projected/748dd79d-83b4-47cb-8282-847f3f4fa3de-kube-api-access-br6f2\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.086182 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.086165    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-token\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.086218 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.086192    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-podinfo\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.086218 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.086212    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-workload-certs\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.086325 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.086311    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748dd79d-83b4-47cb-8282-847f3f4fa3de-istiod-ca-cert\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.086363 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.086342    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-data\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.086415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.086404    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-credential-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.128832 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.128800    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:43.187726 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.187523    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br6f2\" (UniqueName: \"kubernetes.io/projected/748dd79d-83b4-47cb-8282-847f3f4fa3de-kube-api-access-br6f2\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.187726 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.187704    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-token\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.188061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.187755    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-podinfo\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.188061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.187830    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-workload-certs\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.188061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.187892    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748dd79d-83b4-47cb-8282-847f3f4fa3de-istiod-ca-cert\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.188061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.187933    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-data\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.188061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.188008    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-credential-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.188061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.188058    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-envoy\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.188377 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.188118    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-workload-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.188471 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.188443    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-data\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.188671 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.188649    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-credential-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.188886 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.188675    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-workload-certs\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.189034 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.188698    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-workload-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.189163 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.189141    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748dd79d-83b4-47cb-8282-847f3f4fa3de-istiod-ca-cert\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.204974 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.204937    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-envoy\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.206297 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.206200    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-podinfo\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.208409 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.207701    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" event={"ID":"2323f996-abc1-4f51-9ed6-2e96cde226bc","Type":"ContainerStarted","Data":"0a910cf45bf32f35bde6162a6c65a2c0ee7d18a16e5bd318481db7136ca2c1cc"}
Oct 31 04:01:43.263668 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.263599    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-token\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.265936 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.265858    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 04:01:43.270446 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.269959    4900 patch_prober.go:29] interesting pod/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 04:01:43.270446 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.270030    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 04:01:43.271356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.271327    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br6f2\" (UniqueName: \"kubernetes.io/projected/748dd79d-83b4-47cb-8282-847f3f4fa3de-kube-api-access-br6f2\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.349443 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.349408    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:43.373414 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.373387    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-5c0940"
Oct 31 04:01:43.492207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.492177    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5a7ea61-b607-4436-85e8-c6f6b85fe31f-kube-api-access-x4clp\" (UniqueName: \"kubernetes.io/projected/b5a7ea61-b607-4436-85e8-c6f6b85fe31f-kube-api-access-x4clp\") pod \"b5a7ea61-b607-4436-85e8-c6f6b85fe31f\" (UID: \"b5a7ea61-b607-4436-85e8-c6f6b85fe31f\") "
Oct 31 04:01:43.494218 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.494192    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5a7ea61-b607-4436-85e8-c6f6b85fe31f-kube-api-access-x4clp" pod "b5a7ea61-b607-4436-85e8-c6f6b85fe31f" (UID: "b5a7ea61-b607-4436-85e8-c6f6b85fe31f"). InnerVolumeSpecName "kube-api-access-x4clp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:01:43.569444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.569407    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:43.593444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:43.593411    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4clp\" (UniqueName: \"kubernetes.io/projected/b5a7ea61-b607-4436-85e8-c6f6b85fe31f-kube-api-access-x4clp\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:01:44.064515 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.064478    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"]
Oct 31 04:01:44.071770 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:01:44.071742    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748dd79d_83b4_47cb_8282_847f3f4fa3de.slice/crio-24a6a3964dca8c66464886105537dd15ae91f567f85e42a953e18c6af94297bc WatchSource:0}: Error finding container 24a6a3964dca8c66464886105537dd15ae91f567f85e42a953e18c6af94297bc: Status 404 returned error can't find the container with id 24a6a3964dca8c66464886105537dd15ae91f567f85e42a953e18c6af94297bc
Oct 31 04:01:44.074432 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.074400    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:01:44.074521 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.074471    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:01:44.074521 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.074502    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:01:44.217431 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.217394    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln" event={"ID":"748dd79d-83b4-47cb-8282-847f3f4fa3de","Type":"ContainerStarted","Data":"d8edb4fa01bd797c1aff6d8568987c8a9f0911bc6f909c8e9788c03ba21ca22c"}
Oct 31 04:01:44.217620 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.217451    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln" event={"ID":"748dd79d-83b4-47cb-8282-847f3f4fa3de","Type":"ContainerStarted","Data":"24a6a3964dca8c66464886105537dd15ae91f567f85e42a953e18c6af94297bc"}
Oct 31 04:01:44.218924 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.218894    4900 generic.go:356] "Generic (PLEG): container finished" podID="b5a7ea61-b607-4436-85e8-c6f6b85fe31f" containerID="efd4cf2d6e5247171e94123c46d612f895127dec9119204f97cb878428fca26f" exitCode=137
Oct 31 04:01:44.219068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.218923    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7205-5c0940"
Oct 31 04:01:44.219068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.218986    4900 scope.go:122] "RemoveContainer" containerID="efd4cf2d6e5247171e94123c46d612f895127dec9119204f97cb878428fca26f"
Oct 31 04:01:44.229431 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.229414    4900 scope.go:122] "RemoveContainer" containerID="efd4cf2d6e5247171e94123c46d612f895127dec9119204f97cb878428fca26f"
Oct 31 04:01:44.229732 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:44.229709    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd4cf2d6e5247171e94123c46d612f895127dec9119204f97cb878428fca26f\": container with ID starting with efd4cf2d6e5247171e94123c46d612f895127dec9119204f97cb878428fca26f not found: ID does not exist" containerID="efd4cf2d6e5247171e94123c46d612f895127dec9119204f97cb878428fca26f"
Oct 31 04:01:44.229823 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.229742    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd4cf2d6e5247171e94123c46d612f895127dec9119204f97cb878428fca26f"} err="failed to get container status \"efd4cf2d6e5247171e94123c46d612f895127dec9119204f97cb878428fca26f\": rpc error: code = NotFound desc = could not find container \"efd4cf2d6e5247171e94123c46d612f895127dec9119204f97cb878428fca26f\": container with ID starting with efd4cf2d6e5247171e94123c46d612f895127dec9119204f97cb878428fca26f not found: ID does not exist"
Oct 31 04:01:44.265965 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.265929    4900 patch_prober.go:29] interesting pod/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 04:01:44.266139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.265995    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 04:01:44.356867 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.356052    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln" podStartSLOduration=2.356028576 podStartE2EDuration="2.356028576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:44.340069875 +0000 UTC m=+3257.306403234" watchObservedRunningTime="2026-10-31 04:01:44.356028576 +0000 UTC m=+3257.322361941"
Oct 31 04:01:44.391094 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:01:44.391050    4900 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7205-5c0940\" is forbidden: User \"system:node:ip-10-0-134-83.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-83.us-west-2.compute.internal' and this object" podUID="b5a7ea61-b607-4436-85e8-c6f6b85fe31f" pod="opendatahub/test-curl-7205-5c0940"
Oct 31 04:01:44.569697 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:44.569661    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:45.165744 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:45.165711    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5a7ea61-b607-4436-85e8-c6f6b85fe31f" path="/var/lib/kubelet/pods/b5a7ea61-b607-4436-85e8-c6f6b85fe31f/volumes"
Oct 31 04:01:45.265548 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:45.265511    4900 patch_prober.go:29] interesting pod/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 04:01:45.265757 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:45.265569    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 04:01:45.575867 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:45.575836    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:46.231286 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:46.231255    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:46.232381 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:46.232358    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:01:46.265330 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:46.265294    4900 patch_prober.go:29] interesting pod/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 04:01:46.265521 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:46.265352    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 04:01:46.960216 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:46.960174    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:47.265331 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:47.265296    4900 patch_prober.go:29] interesting pod/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 04:01:47.265717 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:47.265351    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 04:01:47.430952 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:47.430921    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:47.695763 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:47.695680    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:48.265263 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:48.265231    4900 patch_prober.go:29] interesting pod/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 31 04:01:48.265462 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:48.265304    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 31 04:01:49.270374 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:49.270345    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 04:01:49.270806 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:49.270742    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 04:01:49.271471 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:49.271449    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 04:01:50.686545 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:50.686510    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:50.917478 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:50.917436    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:51.173356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.173318    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"]
Oct 31 04:01:51.182977 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.182946    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"
Oct 31 04:01:51.207420 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.207390    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e628d97-133e-4c25-b9b4-99346b0e6069-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk\" (UID: \"4e628d97-133e-4c25-b9b4-99346b0e6069\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"
Oct 31 04:01:51.207588 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.207529    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngczd\" (UniqueName: \"kubernetes.io/projected/4e628d97-133e-4c25-b9b4-99346b0e6069-kube-api-access-ngczd\") pod \"payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk\" (UID: \"4e628d97-133e-4c25-b9b4-99346b0e6069\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"
Oct 31 04:01:51.211169 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.211146    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"]
Oct 31 04:01:51.258040 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.258005    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c7585c\""
Oct 31 04:01:51.258201 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.258048    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c7585c-dockercfg-mcvcs\""
Oct 31 04:01:51.309193 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.309157    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e628d97-133e-4c25-b9b4-99346b0e6069-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk\" (UID: \"4e628d97-133e-4c25-b9b4-99346b0e6069\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"
Oct 31 04:01:51.309368 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.309227    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngczd\" (UniqueName: \"kubernetes.io/projected/4e628d97-133e-4c25-b9b4-99346b0e6069-kube-api-access-ngczd\") pod \"payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk\" (UID: \"4e628d97-133e-4c25-b9b4-99346b0e6069\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"
Oct 31 04:01:51.309898 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.309874    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e628d97-133e-4c25-b9b4-99346b0e6069-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk\" (UID: \"4e628d97-133e-4c25-b9b4-99346b0e6069\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"
Oct 31 04:01:51.329983 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.329952    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"]
Oct 31 04:01:51.334466 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.334446    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"
Oct 31 04:01:51.366705 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.366679    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngczd\" (UniqueName: \"kubernetes.io/projected/4e628d97-133e-4c25-b9b4-99346b0e6069-kube-api-access-ngczd\") pod \"payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk\" (UID: \"4e628d97-133e-4c25-b9b4-99346b0e6069\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"
Oct 31 04:01:51.386113 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.386087    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"]
Oct 31 04:01:51.409912 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.409883    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6589f-a3bc-43dc-b642-a93a09016021-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z\" (UID: \"bbc6589f-a3bc-43dc-b642-a93a09016021\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"
Oct 31 04:01:51.410047 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.409962    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7gm8\" (UniqueName: \"kubernetes.io/projected/bbc6589f-a3bc-43dc-b642-a93a09016021-kube-api-access-d7gm8\") pod \"payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z\" (UID: \"bbc6589f-a3bc-43dc-b642-a93a09016021\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"
Oct 31 04:01:51.497317 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.497284    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"
Oct 31 04:01:51.510968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.510934    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6589f-a3bc-43dc-b642-a93a09016021-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z\" (UID: \"bbc6589f-a3bc-43dc-b642-a93a09016021\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"
Oct 31 04:01:51.511067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.511043    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7gm8\" (UniqueName: \"kubernetes.io/projected/bbc6589f-a3bc-43dc-b642-a93a09016021-kube-api-access-d7gm8\") pod \"payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z\" (UID: \"bbc6589f-a3bc-43dc-b642-a93a09016021\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"
Oct 31 04:01:51.511886 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.511865    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6589f-a3bc-43dc-b642-a93a09016021-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z\" (UID: \"bbc6589f-a3bc-43dc-b642-a93a09016021\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"
Oct 31 04:01:51.571510 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.571444    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7gm8\" (UniqueName: \"kubernetes.io/projected/bbc6589f-a3bc-43dc-b642-a93a09016021-kube-api-access-d7gm8\") pod \"payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z\" (UID: \"bbc6589f-a3bc-43dc-b642-a93a09016021\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"
Oct 31 04:01:51.647974 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.647944    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"
Oct 31 04:01:51.864133 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:01:51.864097    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e628d97_133e_4c25_b9b4_99346b0e6069.slice/crio-51e50409715cbafc99b945142db687a49d12755d5d46ff1d35316bc2f0767699 WatchSource:0}: Error finding container 51e50409715cbafc99b945142db687a49d12755d5d46ff1d35316bc2f0767699: Status 404 returned error can't find the container with id 51e50409715cbafc99b945142db687a49d12755d5d46ff1d35316bc2f0767699
Oct 31 04:01:51.867121 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:51.867096    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"]
Oct 31 04:01:52.018783 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:01:52.018749    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc6589f_a3bc_43dc_b642_a93a09016021.slice/crio-5a0e9bcd0da1d3786e7f11e41f0a7805cd898766aa19de154c04effe3586ddd9 WatchSource:0}: Error finding container 5a0e9bcd0da1d3786e7f11e41f0a7805cd898766aa19de154c04effe3586ddd9: Status 404 returned error can't find the container with id 5a0e9bcd0da1d3786e7f11e41f0a7805cd898766aa19de154c04effe3586ddd9
Oct 31 04:01:52.019375 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:52.019288    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"]
Oct 31 04:01:52.118137 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:52.118056    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:52.295880 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:52.295845    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk" event={"ID":"4e628d97-133e-4c25-b9b4-99346b0e6069","Type":"ContainerStarted","Data":"f1434cc2d49474bdd5ee5260237d2cd3aea28c9b2efb2d285303c66adb5cee79"}
Oct 31 04:01:52.296086 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:52.295902    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk" event={"ID":"4e628d97-133e-4c25-b9b4-99346b0e6069","Type":"ContainerStarted","Data":"51e50409715cbafc99b945142db687a49d12755d5d46ff1d35316bc2f0767699"}
Oct 31 04:01:52.296086 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:52.296024    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"
Oct 31 04:01:52.297490 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:52.297466    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z" event={"ID":"bbc6589f-a3bc-43dc-b642-a93a09016021","Type":"ContainerStarted","Data":"149218fa72bedc1399a6c7526c54320581fa221bc4d909b78c6cb0b371ce8d4d"}
Oct 31 04:01:52.297621 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:52.297509    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z" event={"ID":"bbc6589f-a3bc-43dc-b642-a93a09016021","Type":"ContainerStarted","Data":"5a0e9bcd0da1d3786e7f11e41f0a7805cd898766aa19de154c04effe3586ddd9"}
Oct 31 04:01:52.297621 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:52.297534    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"
Oct 31 04:01:52.411974 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:52.411868    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk" podStartSLOduration=1.411852802 podStartE2EDuration="1.411852802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:52.409958132 +0000 UTC m=+3265.376291476" watchObservedRunningTime="2026-10-31 04:01:52.411852802 +0000 UTC m=+3265.378186146"
Oct 31 04:01:52.525006 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:52.524942    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z" podStartSLOduration=1.524895049 podStartE2EDuration="1.524895049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:52.522382948 +0000 UTC m=+3265.488716304" watchObservedRunningTime="2026-10-31 04:01:52.524895049 +0000 UTC m=+3265.491228391"
Oct 31 04:01:53.817300 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:53.817263    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:56.928722 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:56.928677    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:57.654594 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:57.654547    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:01:57.955930 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:01:57.955830    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:00.631079 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:00.631023    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:02.222066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:02.222017    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:03.306766 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:03.306739    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"
Oct 31 04:02:03.307171 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:03.306830    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"
Oct 31 04:02:06.610484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:06.610442    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:06.987681 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:02:06.987643    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:02:11.429391 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:11.429357    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:16.467473 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:16.467406    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:18.068411 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:18.068369    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:21.384136 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:21.384055    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:22.098620 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:22.098580    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:23.098487 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.098453    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"]
Oct 31 04:02:23.102440 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.102416    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"
Oct 31 04:02:23.155663 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.155629    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"]
Oct 31 04:02:23.197435 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.197389    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfqwj\" (UniqueName: \"kubernetes.io/projected/2b0a24fa-1f4d-440c-9d62-77182debb7e6-kube-api-access-xfqwj\") pod \"maas-api-revoke-sub-314fc8fe-thbgn\" (UID: \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"
Oct 31 04:02:23.197647 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.197595    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0a24fa-1f4d-440c-9d62-77182debb7e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-314fc8fe-thbgn\" (UID: \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"
Oct 31 04:02:23.197742 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.197703    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0a24fa-1f4d-440c-9d62-77182debb7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-314fc8fe-thbgn\" (UID: \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"
Oct 31 04:02:23.201128 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.201098    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:02:23.201305 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.201211    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:02:23.207492 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.207456    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"]
Oct 31 04:02:23.299097 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.299052    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfqwj\" (UniqueName: \"kubernetes.io/projected/2b0a24fa-1f4d-440c-9d62-77182debb7e6-kube-api-access-xfqwj\") pod \"maas-api-revoke-sub-314fc8fe-thbgn\" (UID: \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"
Oct 31 04:02:23.299354 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.299264    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0a24fa-1f4d-440c-9d62-77182debb7e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-314fc8fe-thbgn\" (UID: \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"
Oct 31 04:02:23.299436 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.299369    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0a24fa-1f4d-440c-9d62-77182debb7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-314fc8fe-thbgn\" (UID: \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"
Oct 31 04:02:23.300350 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.300320    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0a24fa-1f4d-440c-9d62-77182debb7e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-314fc8fe-thbgn\" (UID: \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"
Oct 31 04:02:23.302982 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.302960    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0a24fa-1f4d-440c-9d62-77182debb7e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-314fc8fe-thbgn\" (UID: \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"
Oct 31 04:02:23.327691 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.327652    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:23.355975 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.355905    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfqwj\" (UniqueName: \"kubernetes.io/projected/2b0a24fa-1f4d-440c-9d62-77182debb7e6-kube-api-access-xfqwj\") pod \"maas-api-revoke-sub-314fc8fe-thbgn\" (UID: \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"
Oct 31 04:02:23.416883 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.416850    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"
Oct 31 04:02:23.792337 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:02:23.792302    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0a24fa_1f4d_440c_9d62_77182debb7e6.slice/crio-941e6dfa716445f4ca36229116b689338d667cc196003b1bd71b5342e520f2d9 WatchSource:0}: Error finding container 941e6dfa716445f4ca36229116b689338d667cc196003b1bd71b5342e520f2d9: Status 404 returned error can't find the container with id 941e6dfa716445f4ca36229116b689338d667cc196003b1bd71b5342e520f2d9
Oct 31 04:02:23.794441 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:23.794419    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"]
Oct 31 04:02:24.499445 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:24.499409    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn" event={"ID":"2b0a24fa-1f4d-440c-9d62-77182debb7e6","Type":"ContainerStarted","Data":"9792c7739a89c6672ad0cc3e2719de889e6a7335981efdd5b34f752c81f529fd"}
Oct 31 04:02:24.499879 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:24.499461    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn" event={"ID":"2b0a24fa-1f4d-440c-9d62-77182debb7e6","Type":"ContainerStarted","Data":"941e6dfa716445f4ca36229116b689338d667cc196003b1bd71b5342e520f2d9"}
Oct 31 04:02:24.614067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:24.614021    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn" podStartSLOduration=1.614005648 podStartE2EDuration="1.614005648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:02:24.611649452 +0000 UTC m=+3297.577982797" watchObservedRunningTime="2026-10-31 04:02:24.614005648 +0000 UTC m=+3297.580338992"
Oct 31 04:02:25.096233 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.096197    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:25.510014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.509977    4900 generic.go:356] "Generic (PLEG): container finished" podID="2b0a24fa-1f4d-440c-9d62-77182debb7e6" containerID="9792c7739a89c6672ad0cc3e2719de889e6a7335981efdd5b34f752c81f529fd" exitCode=0
Oct 31 04:02:25.510492 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.510082    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn" event={"ID":"2b0a24fa-1f4d-440c-9d62-77182debb7e6","Type":"ContainerDied","Data":"9792c7739a89c6672ad0cc3e2719de889e6a7335981efdd5b34f752c81f529fd"}
Oct 31 04:02:25.717035 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.717002    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"]
Oct 31 04:02:25.721115 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.721096    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.774242 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.774164    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-e8fa21-openshift-default-dockercfg-4sm4p\""
Oct 31 04:02:25.776541 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.776507    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"]
Oct 31 04:02:25.832279 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.832237    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-workload-socket\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.832447 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.832308    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-podinfo\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.832447 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.832334    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ddz6\" (UniqueName: \"kubernetes.io/projected/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-kube-api-access-6ddz6\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.832447 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.832394    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-data\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.832984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.832481    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-workload-certs\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.832984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.832531    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-token\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.832984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.832555    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istiod-ca-cert\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.832984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.832596    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-credential-socket\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.832984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.832660    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-envoy\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.899410 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.899364    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:25.933919 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.933885    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-workload-socket\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.934122 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.933939    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-podinfo\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.934122 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.933968    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ddz6\" (UniqueName: \"kubernetes.io/projected/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-kube-api-access-6ddz6\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.934122 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.934022    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-data\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.934122 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.934100    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-workload-certs\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.934363 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.934138    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-token\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.934363 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.934173    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istiod-ca-cert\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.934363 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.934209    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-credential-socket\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.934363 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.934257    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-envoy\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.934363 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.934351    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-workload-socket\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.934637 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.934513    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-workload-certs\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.934700 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.934670    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-credential-socket\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.934759 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.934715    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-data\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.935010 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.934989    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istiod-ca-cert\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.936490 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.936469    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-envoy\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.936881 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.936861    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-podinfo\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.992023 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.991986    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-token\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:25.992452 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:25.992429    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ddz6\" (UniqueName: \"kubernetes.io/projected/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-kube-api-access-6ddz6\") pod \"e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") " pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:26.034567 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.034476    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:26.183880 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.183844    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:26.462721 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.462638    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"]
Oct 31 04:02:26.495138 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:02:26.495106    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9fad7ef_4579_40d5_a7a2_e841c2a98bb7.slice/crio-97d33cb294f74b759440a6b2c1397e7e6e064393c16cec90fccc834800cd045f WatchSource:0}: Error finding container 97d33cb294f74b759440a6b2c1397e7e6e064393c16cec90fccc834800cd045f: Status 404 returned error can't find the container with id 97d33cb294f74b759440a6b2c1397e7e6e064393c16cec90fccc834800cd045f
Oct 31 04:02:26.497885 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.497848    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:02:26.498003 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.497934    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:02:26.498003 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.497979    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:02:26.518507 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.518479    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9" event={"ID":"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7","Type":"ContainerStarted","Data":"97d33cb294f74b759440a6b2c1397e7e6e064393c16cec90fccc834800cd045f"}
Oct 31 04:02:26.649087 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.649052    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:26.651884 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.651863    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"
Oct 31 04:02:26.743401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.743370    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b0a24fa-1f4d-440c-9d62-77182debb7e6-kube-api-access-xfqwj\" (UniqueName: \"kubernetes.io/projected/2b0a24fa-1f4d-440c-9d62-77182debb7e6-kube-api-access-xfqwj\") pod \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\" (UID: \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\") "
Oct 31 04:02:26.743582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.743570    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b0a24fa-1f4d-440c-9d62-77182debb7e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0a24fa-1f4d-440c-9d62-77182debb7e6-maas-api-openshift-ca\") pod \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\" (UID: \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\") "
Oct 31 04:02:26.743635 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.743596    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b0a24fa-1f4d-440c-9d62-77182debb7e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0a24fa-1f4d-440c-9d62-77182debb7e6-maas-api-xks-ca\") pod \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\" (UID: \"2b0a24fa-1f4d-440c-9d62-77182debb7e6\") "
Oct 31 04:02:26.743972 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.743942    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0a24fa-1f4d-440c-9d62-77182debb7e6-maas-api-openshift-ca" pod "2b0a24fa-1f4d-440c-9d62-77182debb7e6" (UID: "2b0a24fa-1f4d-440c-9d62-77182debb7e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:02:26.745532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.745500    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0a24fa-1f4d-440c-9d62-77182debb7e6-kube-api-access-xfqwj" pod "2b0a24fa-1f4d-440c-9d62-77182debb7e6" (UID: "2b0a24fa-1f4d-440c-9d62-77182debb7e6"). InnerVolumeSpecName "kube-api-access-xfqwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:26.745907 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.745887    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0a24fa-1f4d-440c-9d62-77182debb7e6-maas-api-xks-ca" pod "2b0a24fa-1f4d-440c-9d62-77182debb7e6" (UID: "2b0a24fa-1f4d-440c-9d62-77182debb7e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:02:26.845144 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.845110    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0a24fa-1f4d-440c-9d62-77182debb7e6-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:26.845144 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.845142    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0a24fa-1f4d-440c-9d62-77182debb7e6-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:26.845144 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:26.845152    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfqwj\" (UniqueName: \"kubernetes.io/projected/2b0a24fa-1f4d-440c-9d62-77182debb7e6-kube-api-access-xfqwj\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:27.526283 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:27.526240    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn" event={"ID":"2b0a24fa-1f4d-440c-9d62-77182debb7e6","Type":"ContainerDied","Data":"941e6dfa716445f4ca36229116b689338d667cc196003b1bd71b5342e520f2d9"}
Oct 31 04:02:27.526716 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:27.526260    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"
Oct 31 04:02:27.526716 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:27.526345    4900 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="941e6dfa716445f4ca36229116b689338d667cc196003b1bd71b5342e520f2d9"
Oct 31 04:02:27.528023 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:27.527984    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9" event={"ID":"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7","Type":"ContainerStarted","Data":"8490b1f2c2943ab88196226ae0ed530ab326571f248bc85000f074e110f35777"}
Oct 31 04:02:27.736464 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:27.736410    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9" podStartSLOduration=2.736391543 podStartE2EDuration="2.736391543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:02:27.695530762 +0000 UTC m=+3300.661864107" watchObservedRunningTime="2026-10-31 04:02:27.736391543 +0000 UTC m=+3300.702724961"
Oct 31 04:02:27.814257 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:27.814173    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:28.035583 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:28.035547    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:28.041554 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:28.041524    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:28.533673 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:28.533651    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:28.534773 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:28.534754    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:02:28.774504 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:28.774420    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:02:28.796855 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:28.796825    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:02:30.673576 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:30.673540    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:30.909132 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:30.909095    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:31.830108 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:31.830075    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:34.092102 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:34.092055    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:34.328399 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:34.328360    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:36.983731 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:02:36.983695    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:02:37.288481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:37.288400    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:38.228043 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.228006    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj" podUID="d682228b-0707-4cf7-9783-79726934b6f9" containerName="main" containerID="cri-o://2b3c587b3f5f2be7d3cf3319d907133bdce5407650f1126bce8ed4b6f36d691d" gracePeriod=45
Oct 31 04:02:38.494679 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.494653    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 04:02:38.593594 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.593561    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-dshm\") pod \"d682228b-0707-4cf7-9783-79726934b6f9\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") "
Oct 31 04:02:38.593777 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.593659    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-tmp-dir\") pod \"d682228b-0707-4cf7-9783-79726934b6f9\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") "
Oct 31 04:02:38.593777 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.593684    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d682228b-0707-4cf7-9783-79726934b6f9-kube-api-access-swnrg\" (UniqueName: \"kubernetes.io/projected/d682228b-0707-4cf7-9783-79726934b6f9-kube-api-access-swnrg\") pod \"d682228b-0707-4cf7-9783-79726934b6f9\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") "
Oct 31 04:02:38.593777 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.593769    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-home\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-home\") pod \"d682228b-0707-4cf7-9783-79726934b6f9\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") "
Oct 31 04:02:38.593984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.593814    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d682228b-0707-4cf7-9783-79726934b6f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/d682228b-0707-4cf7-9783-79726934b6f9-tls-certs\") pod \"d682228b-0707-4cf7-9783-79726934b6f9\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") "
Oct 31 04:02:38.593984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.593874    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-model-cache\") pod \"d682228b-0707-4cf7-9783-79726934b6f9\" (UID: \"d682228b-0707-4cf7-9783-79726934b6f9\") "
Oct 31 04:02:38.593984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.593960    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-tmp-dir" pod "d682228b-0707-4cf7-9783-79726934b6f9" (UID: "d682228b-0707-4cf7-9783-79726934b6f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:38.594140 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.594079    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-home" pod "d682228b-0707-4cf7-9783-79726934b6f9" (UID: "d682228b-0707-4cf7-9783-79726934b6f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:38.594227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.594211    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-model-cache" pod "d682228b-0707-4cf7-9783-79726934b6f9" (UID: "d682228b-0707-4cf7-9783-79726934b6f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:38.594423 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.594408    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:38.594476 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.594425    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:38.594476 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.594434    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:38.596229 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.596200    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d682228b-0707-4cf7-9783-79726934b6f9-tls-certs" pod "d682228b-0707-4cf7-9783-79726934b6f9" (UID: "d682228b-0707-4cf7-9783-79726934b6f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:02:38.596229 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.596220    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d682228b-0707-4cf7-9783-79726934b6f9-kube-api-access-swnrg" pod "d682228b-0707-4cf7-9783-79726934b6f9" (UID: "d682228b-0707-4cf7-9783-79726934b6f9"). InnerVolumeSpecName "kube-api-access-swnrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:38.596393 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.596263    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-dshm" pod "d682228b-0707-4cf7-9783-79726934b6f9" (UID: "d682228b-0707-4cf7-9783-79726934b6f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:38.596937 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.596910    4900 generic.go:356] "Generic (PLEG): container finished" podID="d682228b-0707-4cf7-9783-79726934b6f9" containerID="2b3c587b3f5f2be7d3cf3319d907133bdce5407650f1126bce8ed4b6f36d691d" exitCode=0
Oct 31 04:02:38.597040 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.596959    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"
Oct 31 04:02:38.597040 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.596993    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj" event={"ID":"d682228b-0707-4cf7-9783-79726934b6f9","Type":"ContainerDied","Data":"2b3c587b3f5f2be7d3cf3319d907133bdce5407650f1126bce8ed4b6f36d691d"}
Oct 31 04:02:38.597114 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.597039    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj" event={"ID":"d682228b-0707-4cf7-9783-79726934b6f9","Type":"ContainerDied","Data":"cef28a435d93afb7994b85bb3fc466211e265fe655a3b8be36d6434352c5a582"}
Oct 31 04:02:38.597114 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.597062    4900 scope.go:122] "RemoveContainer" containerID="2b3c587b3f5f2be7d3cf3319d907133bdce5407650f1126bce8ed4b6f36d691d"
Oct 31 04:02:38.627428 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.627402    4900 scope.go:122] "RemoveContainer" containerID="2b3c587b3f5f2be7d3cf3319d907133bdce5407650f1126bce8ed4b6f36d691d"
Oct 31 04:02:38.627742 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:02:38.627715    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3c587b3f5f2be7d3cf3319d907133bdce5407650f1126bce8ed4b6f36d691d\": container with ID starting with 2b3c587b3f5f2be7d3cf3319d907133bdce5407650f1126bce8ed4b6f36d691d not found: ID does not exist" containerID="2b3c587b3f5f2be7d3cf3319d907133bdce5407650f1126bce8ed4b6f36d691d"
Oct 31 04:02:38.627820 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.627742    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3c587b3f5f2be7d3cf3319d907133bdce5407650f1126bce8ed4b6f36d691d"} err="failed to get container status \"2b3c587b3f5f2be7d3cf3319d907133bdce5407650f1126bce8ed4b6f36d691d\": rpc error: code = NotFound desc = could not find container \"2b3c587b3f5f2be7d3cf3319d907133bdce5407650f1126bce8ed4b6f36d691d\": container with ID starting with 2b3c587b3f5f2be7d3cf3319d907133bdce5407650f1126bce8ed4b6f36d691d not found: ID does not exist"
Oct 31 04:02:38.695369 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.695339    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swnrg\" (UniqueName: \"kubernetes.io/projected/d682228b-0707-4cf7-9783-79726934b6f9-kube-api-access-swnrg\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:38.695369 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.695364    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d682228b-0707-4cf7-9783-79726934b6f9-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:38.695369 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.695375    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d682228b-0707-4cf7-9783-79726934b6f9-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:38.772312 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.772234    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"]
Oct 31 04:02:38.775690 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:38.775663    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-c0d50c/auth-test-model-c0d50c-kserve-5bcbf5b8fc-ksbmj"]
Oct 31 04:02:39.153569 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:39.153484    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d682228b-0707-4cf7-9783-79726934b6f9" path="/var/lib/kubelet/pods/d682228b-0707-4cf7-9783-79726934b6f9/volumes"
Oct 31 04:02:42.491849 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:42.491813    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:47.714843 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:47.714810    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:51.923528 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:51.923490    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:53.320237 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:53.320206    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:53.544131 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:53.544102    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:53.760706 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:53.760674    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:54.216201 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:54.216121    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:55.053215 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:55.053182    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:58.976672 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:58.976641    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:02:59.189157 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:02:59.189123    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:00.901249 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:00.901217    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:01.131587 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:01.131554    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:01.354390 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:01.354361    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:01.582016 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:01.581986    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:04.355869 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:04.355837    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:06.325230 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:06.325192    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:06.984044 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:03:06.984008    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:03:08.974102 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:08.974068    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:10.973208 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:10.973177    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:12.144570 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:12.144538    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:12.465687 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:12.465593    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:13.694154 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:13.694121    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:14.208525 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:14.208487    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:15.060309 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:15.060278    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:15.677929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:15.677897    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:15.904212 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:15.904177    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:16.461949 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:16.461914    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:20.671082 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:20.671043    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:20.945866 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:20.945750    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:21.891179 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:21.891147    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:23.118207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.118173    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:23.159929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.159902    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"]
Oct 31 04:03:23.160507 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.160488    4900 cpu_manager.go:414] "containerMap: removing container" podUID="d682228b-0707-4cf7-9783-79726934b6f9" containerName="main"
Oct 31 04:03:23.160507 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.160507    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d682228b-0707-4cf7-9783-79726934b6f9" containerName="main"
Oct 31 04:03:23.160664 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.160521    4900 cpu_manager.go:414] "containerMap: removing container" podUID="2b0a24fa-1f4d-440c-9d62-77182debb7e6" containerName="revoke-keys"
Oct 31 04:03:23.160664 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.160526    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0a24fa-1f4d-440c-9d62-77182debb7e6" containerName="revoke-keys"
Oct 31 04:03:23.160664 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.160615    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="d682228b-0707-4cf7-9783-79726934b6f9" containerName="main"
Oct 31 04:03:23.160664 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.160631    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0a24fa-1f4d-440c-9d62-77182debb7e6" containerName="revoke-keys"
Oct 31 04:03:23.164227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.164202    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.216952 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.216922    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-856ddc\"/\"openshift-service-ca.crt\""
Oct 31 04:03:23.218051 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.218027    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"]
Oct 31 04:03:23.221600 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.221573    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-dshm\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.221716 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.221660    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-tmp-dir\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.221780 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.221733    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2gr6\" (UniqueName: \"kubernetes.io/projected/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-kube-api-access-b2gr6\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.221976 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.221957    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-tls-certs\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.222063 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.221995    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-home\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.222063 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.222029    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-model-cache\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.264366 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.264342    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-856ddc\"/\"explicit-model-856ddc-kserve-self-signed-certs\""
Oct 31 04:03:23.264469 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.264387    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-856ddc\"/\"kube-root-ca.crt\""
Oct 31 04:03:23.264915 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.264887    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-856ddc\"/\"default-dockercfg-6wmrz\""
Oct 31 04:03:23.323608 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.323573    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-dshm\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.323783 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.323625    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-tmp-dir\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.323783 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.323687    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2gr6\" (UniqueName: \"kubernetes.io/projected/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-kube-api-access-b2gr6\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.323783 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.323781    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-tls-certs\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.323965 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.323833    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-home\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.323965 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.323870    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-model-cache\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.324155 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.324132    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-tmp-dir\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.324224 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.324169    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-home\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.324224 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.324208    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-model-cache\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.326300 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.326262    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-dshm\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.327305 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.327280    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-tls-certs\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.351125 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.351103    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:23.380692 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.380619    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2gr6\" (UniqueName: \"kubernetes.io/projected/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-kube-api-access-b2gr6\") pod \"explicit-model-856ddc-kserve-768d6757c9-bwjt9\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") " pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.476275 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.476249    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:23.842663 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:03:23.842570    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c6d7554_7b9e_4422_baf1_8ff8f2f949b1.slice/crio-6bda1a3c10421e829c68ec1276309537dd292750cbc4beb3ac517e21af00b24f WatchSource:0}: Error finding container 6bda1a3c10421e829c68ec1276309537dd292750cbc4beb3ac517e21af00b24f: Status 404 returned error can't find the container with id 6bda1a3c10421e829c68ec1276309537dd292750cbc4beb3ac517e21af00b24f
Oct 31 04:03:23.845319 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.845294    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"]
Oct 31 04:03:23.882355 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.882321    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9" event={"ID":"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1","Type":"ContainerStarted","Data":"6bda1a3c10421e829c68ec1276309537dd292750cbc4beb3ac517e21af00b24f"}
Oct 31 04:03:23.983173 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:23.983142    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:24.195938 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:24.195855    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:24.892103 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:24.892068    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9" event={"ID":"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1","Type":"ContainerStarted","Data":"4d24f75f8572b70b26547cdb3025d3ca3ad8408f240642169dc69ad733a86453"}
Oct 31 04:03:25.005312 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:25.005260    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9" podStartSLOduration=2.005245837 podStartE2EDuration="2.005245837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:25.004687411 +0000 UTC m=+3357.971020755" watchObservedRunningTime="2026-10-31 04:03:25.005245837 +0000 UTC m=+3357.971579180"
Oct 31 04:03:25.477250 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:25.477218    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:27.355843 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:27.355808    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:31.132700 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:31.132660    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:32.023039 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:32.023003    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:32.366959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:32.366875    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:33.476717 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:33.476682    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:33.489490 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:33.489465    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:33.594317 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:33.594287    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:33.957465 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:33.957424    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:03:34.208997 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:34.208918    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:34.488224 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:34.488195    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:36.985007 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:03:36.984974    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:03:37.190892 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:37.190859    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:38.706621 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:38.706590    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:38.968084 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:38.967831    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:42.028304 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:42.028259    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:42.256683 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:42.256647    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:42.613431 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:42.613377    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:42.683612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:42.683581    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"]
Oct 31 04:03:42.688509 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:42.688488    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"
Oct 31 04:03:42.764723 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:42.764683    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"]
Oct 31 04:03:42.797323 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:42.797293    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52l9b\" (UniqueName: \"kubernetes.io/projected/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631-kube-api-access-52l9b\") pod \"payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g\" (UID: \"2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"
Oct 31 04:03:42.797483 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:42.797435    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g\" (UID: \"2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"
Oct 31 04:03:42.805273 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:42.805250    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d8190b\""
Oct 31 04:03:42.805392 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:42.805305    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d8190b-dockercfg-gj2b6\""
Oct 31 04:03:42.898328 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:42.898248    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g\" (UID: \"2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"
Oct 31 04:03:42.898476 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:42.898413    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52l9b\" (UniqueName: \"kubernetes.io/projected/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631-kube-api-access-52l9b\") pod \"payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g\" (UID: \"2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"
Oct 31 04:03:42.899046 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:42.899029    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g\" (UID: \"2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"
Oct 31 04:03:42.961233 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:42.961201    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52l9b\" (UniqueName: \"kubernetes.io/projected/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631-kube-api-access-52l9b\") pod \"payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g\" (UID: \"2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"
Oct 31 04:03:43.002444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:43.002417    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"
Oct 31 04:03:43.391909 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:03:43.391877    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc5ed9f_360f_4bbd_a1f8_3eea73f2b631.slice/crio-15b7b2efa5497c61be5baca59944d2d67132c249c26aaddaec803c5d58519c2f WatchSource:0}: Error finding container 15b7b2efa5497c61be5baca59944d2d67132c249c26aaddaec803c5d58519c2f: Status 404 returned error can't find the container with id 15b7b2efa5497c61be5baca59944d2d67132c249c26aaddaec803c5d58519c2f
Oct 31 04:03:43.395944 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:43.395918    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"]
Oct 31 04:03:43.964071 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:43.964029    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"]
Oct 31 04:03:43.968856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:43.968830    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.007250 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.007218    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g" event={"ID":"2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631","Type":"ContainerStarted","Data":"cd131b06fe3eb32e79d3ebf7d4cf5d9023c141da2f1b5df48e29a35837304fd4"}
Oct 31 04:03:44.007415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.007270    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g" event={"ID":"2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631","Type":"ContainerStarted","Data":"15b7b2efa5497c61be5baca59944d2d67132c249c26aaddaec803c5d58519c2f"}
Oct 31 04:03:44.007415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.007290    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"
Oct 31 04:03:44.012945 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.012922    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-envoy\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.013043 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.012963    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-workload-certs\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.013089 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.013064    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istiod-ca-cert\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.013127 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.013100    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-data\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.013161 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.013129    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h77gp\" (UniqueName: \"kubernetes.io/projected/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-kube-api-access-h77gp\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.013196 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.013160    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-workload-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.013268 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.013253    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-podinfo\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.013307 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.013298    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-credential-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.013396 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.013385    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-token\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.075073 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.075036    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"]
Oct 31 04:03:44.114231 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.114198    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-credential-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.114415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.114274    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-token\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.114415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.114332    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-envoy\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.114415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.114369    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-workload-certs\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.114415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.114411    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istiod-ca-cert\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.114660 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.114443    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-data\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.114660 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.114464    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h77gp\" (UniqueName: \"kubernetes.io/projected/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-kube-api-access-h77gp\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.114660 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.114483    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-workload-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.114660 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.114528    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-podinfo\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.114923 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.114707    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-credential-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.115214 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.115188    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-data\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.115326 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.115215    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-workload-certs\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.115326 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.115271    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istiod-ca-cert\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.115444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.115427    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-workload-socket\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.117463 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.117432    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-envoy\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.117625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.117611    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-podinfo\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.172058 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.172020    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-token\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.172058 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.172037    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h77gp\" (UniqueName: \"kubernetes.io/projected/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-kube-api-access-h77gp\") pod \"e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") " pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.215811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.215723    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g" podStartSLOduration=2.215710181 podStartE2EDuration="2.215710181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:44.212664524 +0000 UTC m=+3377.178997867" watchObservedRunningTime="2026-10-31 04:03:44.215710181 +0000 UTC m=+3377.182043524"
Oct 31 04:03:44.287104 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.287073    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:44.706358 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:03:44.706326    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c901c9_0bd8_48ad_8642_04e39ba8e66d.slice/crio-94fdbc2bb9c4a7dfaef16082c3937242ab254b54c9f9b118af2682821e9dae22 WatchSource:0}: Error finding container 94fdbc2bb9c4a7dfaef16082c3937242ab254b54c9f9b118af2682821e9dae22: Status 404 returned error can't find the container with id 94fdbc2bb9c4a7dfaef16082c3937242ab254b54c9f9b118af2682821e9dae22
Oct 31 04:03:44.707772 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.707748    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"]
Oct 31 04:03:44.709300 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.709268    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:03:44.709401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.709332    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:03:44.709401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:44.709360    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:03:45.016302 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:45.016261    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh" event={"ID":"a1c901c9-0bd8-48ad-8642-04e39ba8e66d","Type":"ContainerStarted","Data":"f6002fb69cdec7cacbe52e261c4ec49419128231a3322f0f3c1b5f01358a29e2"}
Oct 31 04:03:45.016486 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:45.016324    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh" event={"ID":"a1c901c9-0bd8-48ad-8642-04e39ba8e66d","Type":"ContainerStarted","Data":"94fdbc2bb9c4a7dfaef16082c3937242ab254b54c9f9b118af2682821e9dae22"}
Oct 31 04:03:45.133776 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:45.133725    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh" podStartSLOduration=2.133712335 podStartE2EDuration="2.133712335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:03:45.132338473 +0000 UTC m=+3378.098671818" watchObservedRunningTime="2026-10-31 04:03:45.133712335 +0000 UTC m=+3378.100045678"
Oct 31 04:03:45.287887 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:45.287760    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:45.289296 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:45.289229    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 04:03:45.289296 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:45.289287    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh" podUID="a1c901c9-0bd8-48ad-8642-04e39ba8e66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 04:03:46.287561 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:46.287527    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 04:03:46.287965 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:46.287579    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh" podUID="a1c901c9-0bd8-48ad-8642-04e39ba8e66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 04:03:47.272944 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:47.272901    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:03:47.273124 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:47.272973    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln" podUID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:03:47.288388 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:47.288358    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 04:03:47.288753 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:47.288406    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh" podUID="a1c901c9-0bd8-48ad-8642-04e39ba8e66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 04:03:47.995225 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:47.995193    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:48.218901 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:48.218865    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:48.288155 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:48.288055    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 04:03:48.288155 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:48.288114    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh" podUID="a1c901c9-0bd8-48ad-8642-04e39ba8e66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 04:03:48.459534 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:48.459464    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:49.066481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:49.066450    4900 generic.go:356] "Generic (PLEG): container finished" podID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerID="d8edb4fa01bd797c1aff6d8568987c8a9f0911bc6f909c8e9788c03ba21ca22c" exitCode=0
Oct 31 04:03:49.066705 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:49.066524    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln" event={"ID":"748dd79d-83b4-47cb-8282-847f3f4fa3de","Type":"ContainerDied","Data":"d8edb4fa01bd797c1aff6d8568987c8a9f0911bc6f909c8e9788c03ba21ca22c"}
Oct 31 04:03:49.067219 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:49.067199    4900 scope.go:122] "RemoveContainer" containerID="d8edb4fa01bd797c1aff6d8568987c8a9f0911bc6f909c8e9788c03ba21ca22c"
Oct 31 04:03:49.069351 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:49.069319    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:03:49.069450 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:49.069404    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:03:49.069497 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:49.069448    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:03:49.287370 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:49.287341    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 04:03:49.287564 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:49.287492    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh" podUID="a1c901c9-0bd8-48ad-8642-04e39ba8e66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 04:03:50.094776 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:50.094727    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln" event={"ID":"748dd79d-83b4-47cb-8282-847f3f4fa3de","Type":"ContainerStarted","Data":"a4cbc70c27cb81be2a475175979fe5dac713c9c8a36439a5736113284d975763"}
Oct 31 04:03:50.287818 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:50.287752    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 04:03:50.288122 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:50.288081    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh" podUID="a1c901c9-0bd8-48ad-8642-04e39ba8e66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 04:03:50.571810 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:50.571163    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:03:50.571810 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:50.571290    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 04:03:50.571810 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:50.571334    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln" podUID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 04:03:51.064709 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:51.064671    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:51.293539 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:51.293507    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:51.293935 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:51.293893    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:51.294515 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:51.294495    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:03:51.570754 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:51.570722    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 04:03:51.570974 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:51.570802    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln" podUID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 04:03:52.569935 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:52.569896    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 04:03:52.570370 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:52.569949    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln" podUID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 04:03:53.570340 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:53.570302    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 04:03:53.570765 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:53.570358    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln" podUID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 04:03:53.625613 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:53.625577    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:03:54.570324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:54.570283    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 04:03:54.570512 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:54.570344    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln" podUID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 04:03:55.019402 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:55.019374    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"
Oct 31 04:03:55.573862 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:55.573829    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:03:55.574367 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:55.574174    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:03:55.574872 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:55.574851    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:03:58.795274 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:03:58.795236    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:01.310297 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:01.310259    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:01.575100 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:01.575020    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:02.554504 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:02.554467    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:03.605961 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:03.605929    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:06.986953 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:04:06.986915    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:04:07.436148 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:07.436047    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"]
Oct 31 04:04:07.589804 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:07.589762    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:08.661013 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:08.660968    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:13.515696 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:13.515654    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:17.259776 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:17.259741    4900 generic.go:356] "Generic (PLEG): container finished" podID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" containerID="8490b1f2c2943ab88196226ae0ed530ab326571f248bc85000f074e110f35777" exitCode=0
Oct 31 04:04:17.260258 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:17.259819    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9" event={"ID":"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7","Type":"ContainerDied","Data":"8490b1f2c2943ab88196226ae0ed530ab326571f248bc85000f074e110f35777"}
Oct 31 04:04:17.260303 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:17.260259    4900 scope.go:122] "RemoveContainer" containerID="8490b1f2c2943ab88196226ae0ed530ab326571f248bc85000f074e110f35777"
Oct 31 04:04:17.261867 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:17.261835    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:04:17.261976 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:17.261920    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:04:17.261976 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:17.261970    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:04:18.268196 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:18.268155    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9" event={"ID":"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7","Type":"ContainerStarted","Data":"89d38cb4b524be92408b5629fb387bd7246875250586feb762695f701a5e5995"}
Oct 31 04:04:18.390006 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:18.389975    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:19.054813 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:19.054153    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:04:19.054813 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:19.054306    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 31 04:04:19.054813 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:19.054361    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9" podUID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 31 04:04:20.058978 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:20.058935    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 31 04:04:20.059494 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:20.058999    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9" podUID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 31 04:04:21.059263 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:21.059222    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 31 04:04:21.059777 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:21.059293    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9" podUID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 31 04:04:22.055813 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.055210    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 31 04:04:22.055813 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.055274    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9" podUID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 31 04:04:22.467060 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.466958    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9" podUID="5c6d7554-7b9e-4422-baf1-8ff8f2f949b1" containerName="main" containerID="cri-o://4d24f75f8572b70b26547cdb3025d3ca3ad8408f240642169dc69ad733a86453" gracePeriod=45
Oct 31 04:04:22.832264 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.832235    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:04:22.915279 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.915244    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-tls-certs\") pod \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") "
Oct 31 04:04:22.915482 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.915295    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-home\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-home\") pod \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") "
Oct 31 04:04:22.915482 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.915347    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-kube-api-access-b2gr6\" (UniqueName: \"kubernetes.io/projected/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-kube-api-access-b2gr6\") pod \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") "
Oct 31 04:04:22.915482 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.915400    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-tmp-dir\") pod \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") "
Oct 31 04:04:22.915482 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.915452    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-dshm\") pod \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") "
Oct 31 04:04:22.915712 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.915519    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-model-cache\") pod \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\" (UID: \"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1\") "
Oct 31 04:04:22.915712 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.915642    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-home" pod "5c6d7554-7b9e-4422-baf1-8ff8f2f949b1" (UID: "5c6d7554-7b9e-4422-baf1-8ff8f2f949b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:04:22.915712 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.915661    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-tmp-dir" pod "5c6d7554-7b9e-4422-baf1-8ff8f2f949b1" (UID: "5c6d7554-7b9e-4422-baf1-8ff8f2f949b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:04:22.915894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.915868    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-model-cache" pod "5c6d7554-7b9e-4422-baf1-8ff8f2f949b1" (UID: "5c6d7554-7b9e-4422-baf1-8ff8f2f949b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:04:22.916144 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.916118    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:22.916248 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.916147    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:22.916248 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.916162    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:22.917658 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.917627    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-tls-certs" pod "5c6d7554-7b9e-4422-baf1-8ff8f2f949b1" (UID: "5c6d7554-7b9e-4422-baf1-8ff8f2f949b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:04:22.917752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.917670    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-dshm" pod "5c6d7554-7b9e-4422-baf1-8ff8f2f949b1" (UID: "5c6d7554-7b9e-4422-baf1-8ff8f2f949b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:04:22.917752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:22.917728    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-kube-api-access-b2gr6" pod "5c6d7554-7b9e-4422-baf1-8ff8f2f949b1" (UID: "5c6d7554-7b9e-4422-baf1-8ff8f2f949b1"). InnerVolumeSpecName "kube-api-access-b2gr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:04:23.017931 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.017894    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:23.017931 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.017927    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2gr6\" (UniqueName: \"kubernetes.io/projected/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-kube-api-access-b2gr6\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:23.017931 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.017940    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:23.059920 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.059416    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 31 04:04:23.059920 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.059497    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9" podUID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 31 04:04:23.286011 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.285974    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:23.299733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.299704    4900 generic.go:356] "Generic (PLEG): container finished" podID="5c6d7554-7b9e-4422-baf1-8ff8f2f949b1" containerID="4d24f75f8572b70b26547cdb3025d3ca3ad8408f240642169dc69ad733a86453" exitCode=0
Oct 31 04:04:23.299959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.299761    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"
Oct 31 04:04:23.299959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.299819    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9" event={"ID":"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1","Type":"ContainerDied","Data":"4d24f75f8572b70b26547cdb3025d3ca3ad8408f240642169dc69ad733a86453"}
Oct 31 04:04:23.299959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.299874    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9" event={"ID":"5c6d7554-7b9e-4422-baf1-8ff8f2f949b1","Type":"ContainerDied","Data":"6bda1a3c10421e829c68ec1276309537dd292750cbc4beb3ac517e21af00b24f"}
Oct 31 04:04:23.299959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.299900    4900 scope.go:122] "RemoveContainer" containerID="4d24f75f8572b70b26547cdb3025d3ca3ad8408f240642169dc69ad733a86453"
Oct 31 04:04:23.310031 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.309981    4900 scope.go:122] "RemoveContainer" containerID="4d24f75f8572b70b26547cdb3025d3ca3ad8408f240642169dc69ad733a86453"
Oct 31 04:04:23.310297 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:04:23.310263    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d24f75f8572b70b26547cdb3025d3ca3ad8408f240642169dc69ad733a86453\": container with ID starting with 4d24f75f8572b70b26547cdb3025d3ca3ad8408f240642169dc69ad733a86453 not found: ID does not exist" containerID="4d24f75f8572b70b26547cdb3025d3ca3ad8408f240642169dc69ad733a86453"
Oct 31 04:04:23.310371 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.310298    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d24f75f8572b70b26547cdb3025d3ca3ad8408f240642169dc69ad733a86453"} err="failed to get container status \"4d24f75f8572b70b26547cdb3025d3ca3ad8408f240642169dc69ad733a86453\": rpc error: code = NotFound desc = could not find container \"4d24f75f8572b70b26547cdb3025d3ca3ad8408f240642169dc69ad733a86453\": container with ID starting with 4d24f75f8572b70b26547cdb3025d3ca3ad8408f240642169dc69ad733a86453 not found: ID does not exist"
Oct 31 04:04:23.464739 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.464698    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"]
Oct 31 04:04:23.468603 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:23.468570    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-856ddc/explicit-model-856ddc-kserve-768d6757c9-bwjt9"]
Oct 31 04:04:24.039802 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:24.039756    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:04:24.040298 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:24.040269    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:04:24.041006 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:24.040988    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:04:25.153977 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:25.153945    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6d7554-7b9e-4422-baf1-8ff8f2f949b1" path="/var/lib/kubelet/pods/5c6d7554-7b9e-4422-baf1-8ff8f2f949b1/volumes"
Oct 31 04:04:26.226815 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:26.226771    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:27.554641 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:27.554606    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:28.253338 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:28.253290    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:28.489219 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:28.489185    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:28.700777 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:28.700748    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:29.905529 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:29.905496    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:30.160835 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:30.160726    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:32.177176 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:32.177146    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:32.416058 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:32.416029    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:33.980151 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:33.980115    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:35.428882 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.428852    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:35.653522 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.653487    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:35.757313 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.757277    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"]
Oct 31 04:04:35.757921 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.757906    4900 cpu_manager.go:414] "containerMap: removing container" podUID="5c6d7554-7b9e-4422-baf1-8ff8f2f949b1" containerName="main"
Oct 31 04:04:35.757983 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.757921    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6d7554-7b9e-4422-baf1-8ff8f2f949b1" containerName="main"
Oct 31 04:04:35.758039 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.758031    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6d7554-7b9e-4422-baf1-8ff8f2f949b1" containerName="main"
Oct 31 04:04:35.761413 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.761399    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"
Oct 31 04:04:35.777029 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.777005    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j8qh\" (UniqueName: \"kubernetes.io/projected/ab71a411-994b-41ac-a655-6c69e1be910d-kube-api-access-6j8qh\") pod \"payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg\" (UID: \"ab71a411-994b-41ac-a655-6c69e1be910d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"
Oct 31 04:04:35.777135 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.777059    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab71a411-994b-41ac-a655-6c69e1be910d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg\" (UID: \"ab71a411-994b-41ac-a655-6c69e1be910d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"
Oct 31 04:04:35.816664 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.816628    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"]
Oct 31 04:04:35.863834 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.863802    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-1f1314\""
Oct 31 04:04:35.864042 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.863842    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-1f1314-dockercfg-pmxhp\""
Oct 31 04:04:35.882734 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.882693    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j8qh\" (UniqueName: \"kubernetes.io/projected/ab71a411-994b-41ac-a655-6c69e1be910d-kube-api-access-6j8qh\") pod \"payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg\" (UID: \"ab71a411-994b-41ac-a655-6c69e1be910d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"
Oct 31 04:04:35.882970 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.882815    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab71a411-994b-41ac-a655-6c69e1be910d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg\" (UID: \"ab71a411-994b-41ac-a655-6c69e1be910d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"
Oct 31 04:04:35.883972 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.883943    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab71a411-994b-41ac-a655-6c69e1be910d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg\" (UID: \"ab71a411-994b-41ac-a655-6c69e1be910d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"
Oct 31 04:04:35.939256 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:35.939221    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j8qh\" (UniqueName: \"kubernetes.io/projected/ab71a411-994b-41ac-a655-6c69e1be910d-kube-api-access-6j8qh\") pod \"payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg\" (UID: \"ab71a411-994b-41ac-a655-6c69e1be910d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"
Oct 31 04:04:36.075252 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:36.075162    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"
Oct 31 04:04:36.446183 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:04:36.446147    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab71a411_994b_41ac_a655_6c69e1be910d.slice/crio-5e30bc1333f762030d88617a78324fff93ef69c9c915683618fc0721736de6c6 WatchSource:0}: Error finding container 5e30bc1333f762030d88617a78324fff93ef69c9c915683618fc0721736de6c6: Status 404 returned error can't find the container with id 5e30bc1333f762030d88617a78324fff93ef69c9c915683618fc0721736de6c6
Oct 31 04:04:36.447959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:36.447851    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"]
Oct 31 04:04:36.985084 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:04:36.985050    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:04:37.397071 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:37.396983    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg" event={"ID":"ab71a411-994b-41ac-a655-6c69e1be910d","Type":"ContainerStarted","Data":"0b28f4aa8ae2758eda9118b03e6cbcc2229a92d4ea648cdc98f82655f853899d"}
Oct 31 04:04:37.397071 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:37.397032    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg" event={"ID":"ab71a411-994b-41ac-a655-6c69e1be910d","Type":"ContainerStarted","Data":"5e30bc1333f762030d88617a78324fff93ef69c9c915683618fc0721736de6c6"}
Oct 31 04:04:37.397071 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:37.397069    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"
Oct 31 04:04:37.512814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:37.512750    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg" podStartSLOduration=2.512734624 podStartE2EDuration="2.512734624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:37.512171506 +0000 UTC m=+3430.478504851" watchObservedRunningTime="2026-10-31 04:04:37.512734624 +0000 UTC m=+3430.479067968"
Oct 31 04:04:38.904029 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:38.903989    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"]
Oct 31 04:04:38.908274 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:38.908251    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:38.963376 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:38.963341    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"]
Oct 31 04:04:39.010297 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.010265    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-5901fe-serving-cert\""
Oct 31 04:04:39.021524 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.021450    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlsml\" (UniqueName: \"kubernetes.io/projected/d05dfb40-59b0-4026-941c-6c600605d6fd-kube-api-access-jlsml\") pod \"maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:39.021653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.021620    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05dfb40-59b0-4026-941c-6c600605d6fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:39.021780 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.021764    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:39.021845 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.021819    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:39.122658 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.122621    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05dfb40-59b0-4026-941c-6c600605d6fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:39.122859 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.122741    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:39.122859 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.122759    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:39.122859 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.122824    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlsml\" (UniqueName: \"kubernetes.io/projected/d05dfb40-59b0-4026-941c-6c600605d6fd-kube-api-access-jlsml\") pod \"maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:39.123072 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:04:39.122957    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-serving-cert: secret "maas-api-e2e-shared-a-w0-5901fe-serving-cert" not found
Oct 31 04:04:39.123072 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:04:39.123018    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-maas-api-tls podName:d05dfb40-59b0-4026-941c-6c600605d6fd nodeName:}" failed. No retries permitted until 2026-10-31 04:04:39.623001162 +0000 UTC m=+3432.589334496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-maas-api-tls") pod "maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk" (UID: "d05dfb40-59b0-4026-941c-6c600605d6fd") : secret "maas-api-e2e-shared-a-w0-5901fe-serving-cert" not found
Oct 31 04:04:39.123438 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.123413    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05dfb40-59b0-4026-941c-6c600605d6fd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:39.125290 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.125273    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:39.180346 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.180284    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlsml\" (UniqueName: \"kubernetes.io/projected/d05dfb40-59b0-4026-941c-6c600605d6fd-kube-api-access-jlsml\") pod \"maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:39.187463 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.187439    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:39.401217 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.401185    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:39.629179 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.629143    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:39.631700 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.631675    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:39.822065 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:39.822028    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:40.196653 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:04:40.196606    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05dfb40_59b0_4026_941c_6c600605d6fd.slice/crio-bb9d2281b50ea3b39a34341b9b72986bf7a212cf12510164dbb7d46f5cee7402 WatchSource:0}: Error finding container bb9d2281b50ea3b39a34341b9b72986bf7a212cf12510164dbb7d46f5cee7402: Status 404 returned error can't find the container with id bb9d2281b50ea3b39a34341b9b72986bf7a212cf12510164dbb7d46f5cee7402
Oct 31 04:04:40.199364 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:40.199254    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"]
Oct 31 04:04:40.417536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:40.417500    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk" event={"ID":"d05dfb40-59b0-4026-941c-6c600605d6fd","Type":"ContainerStarted","Data":"bb9d2281b50ea3b39a34341b9b72986bf7a212cf12510164dbb7d46f5cee7402"}
Oct 31 04:04:41.424864 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:41.424755    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk" event={"ID":"d05dfb40-59b0-4026-941c-6c600605d6fd","Type":"ContainerStarted","Data":"984a94abd479889d5da283f760816e3c22d1f1fc18aa91c66bc70270ed337d9a"}
Oct 31 04:04:41.425327 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:41.424862    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:41.541957 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:41.541906    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk" podStartSLOduration=2.613085362 podStartE2EDuration="3.541891499s" totalImagesPullingTime="928.806137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:41.538712634 +0000 UTC m=+3434.505045981" watchObservedRunningTime="2026-10-31 04:04:41.541891499 +0000 UTC m=+3434.508224842"
Oct 31 04:04:44.483325 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:44.483293    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:44.874982 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:44.874873    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:44.894882 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:44.894844    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"]
Oct 31 04:04:45.003183 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:45.003143    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"]
Oct 31 04:04:45.352060 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:45.352014    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:45.565457 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:45.565423    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:47.441026 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:47.441000    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:04:48.405839 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:48.405811    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"
Oct 31 04:04:49.895398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:49.895360    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c" podUID="319c12fa-ccdd-4dac-83e5-802273cc62bf" containerName="payload-processing" containerID="cri-o://41d3e3e464977f3aec61abab5a99417f732da6e032c6e6758683e7b1fd6e2e3a" gracePeriod=25
Oct 31 04:04:50.001333 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.001298    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:50.004175 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.004146    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x" podUID="5420545a-5a77-4a15-bea3-8274d49e5e5d" containerName="payload-pre-processing" containerID="cri-o://da24f71e3cc8c08bf687036e910e8c0fcd308e654ae3c71876b0cd52dc3e1216" gracePeriod=25
Oct 31 04:04:50.177190 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.177113    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"
Oct 31 04:04:50.241649 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.241622    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:50.278744 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.278719    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319c12fa-ccdd-4dac-83e5-802273cc62bf-kube-api-access-rnbkm\" (UniqueName: \"kubernetes.io/projected/319c12fa-ccdd-4dac-83e5-802273cc62bf-kube-api-access-rnbkm\") pod \"319c12fa-ccdd-4dac-83e5-802273cc62bf\" (UID: \"319c12fa-ccdd-4dac-83e5-802273cc62bf\") "
Oct 31 04:04:50.278906 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.278887    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/319c12fa-ccdd-4dac-83e5-802273cc62bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319c12fa-ccdd-4dac-83e5-802273cc62bf-plugins-config-volume\") pod \"319c12fa-ccdd-4dac-83e5-802273cc62bf\" (UID: \"319c12fa-ccdd-4dac-83e5-802273cc62bf\") "
Oct 31 04:04:50.279441 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.279415    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319c12fa-ccdd-4dac-83e5-802273cc62bf-plugins-config-volume" pod "319c12fa-ccdd-4dac-83e5-802273cc62bf" (UID: "319c12fa-ccdd-4dac-83e5-802273cc62bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:04:50.281189 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.281169    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319c12fa-ccdd-4dac-83e5-802273cc62bf-kube-api-access-rnbkm" pod "319c12fa-ccdd-4dac-83e5-802273cc62bf" (UID: "319c12fa-ccdd-4dac-83e5-802273cc62bf"). InnerVolumeSpecName "kube-api-access-rnbkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:04:50.291926 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.291902    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"
Oct 31 04:04:50.380161 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.380125    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5420545a-5a77-4a15-bea3-8274d49e5e5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5420545a-5a77-4a15-bea3-8274d49e5e5d-plugins-config-volume\") pod \"5420545a-5a77-4a15-bea3-8274d49e5e5d\" (UID: \"5420545a-5a77-4a15-bea3-8274d49e5e5d\") "
Oct 31 04:04:50.380369 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.380176    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5420545a-5a77-4a15-bea3-8274d49e5e5d-kube-api-access-7ctvh\" (UniqueName: \"kubernetes.io/projected/5420545a-5a77-4a15-bea3-8274d49e5e5d-kube-api-access-7ctvh\") pod \"5420545a-5a77-4a15-bea3-8274d49e5e5d\" (UID: \"5420545a-5a77-4a15-bea3-8274d49e5e5d\") "
Oct 31 04:04:50.380583 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.380558    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5420545a-5a77-4a15-bea3-8274d49e5e5d-plugins-config-volume" pod "5420545a-5a77-4a15-bea3-8274d49e5e5d" (UID: "5420545a-5a77-4a15-bea3-8274d49e5e5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:04:50.380884 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.380866    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319c12fa-ccdd-4dac-83e5-802273cc62bf-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:50.380962 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.380889    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5420545a-5a77-4a15-bea3-8274d49e5e5d-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:50.380962 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.380903    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnbkm\" (UniqueName: \"kubernetes.io/projected/319c12fa-ccdd-4dac-83e5-802273cc62bf-kube-api-access-rnbkm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:50.382293 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.382273    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5420545a-5a77-4a15-bea3-8274d49e5e5d-kube-api-access-7ctvh" pod "5420545a-5a77-4a15-bea3-8274d49e5e5d" (UID: "5420545a-5a77-4a15-bea3-8274d49e5e5d"). InnerVolumeSpecName "kube-api-access-7ctvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:04:50.482526 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.482488    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ctvh\" (UniqueName: \"kubernetes.io/projected/5420545a-5a77-4a15-bea3-8274d49e5e5d-kube-api-access-7ctvh\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:50.488588 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.488560    4900 generic.go:356] "Generic (PLEG): container finished" podID="319c12fa-ccdd-4dac-83e5-802273cc62bf" containerID="41d3e3e464977f3aec61abab5a99417f732da6e032c6e6758683e7b1fd6e2e3a" exitCode=0
Oct 31 04:04:50.488753 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.488604    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"
Oct 31 04:04:50.488753 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.488687    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c" event={"ID":"319c12fa-ccdd-4dac-83e5-802273cc62bf","Type":"ContainerDied","Data":"41d3e3e464977f3aec61abab5a99417f732da6e032c6e6758683e7b1fd6e2e3a"}
Oct 31 04:04:50.488753 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.488746    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c" event={"ID":"319c12fa-ccdd-4dac-83e5-802273cc62bf","Type":"ContainerDied","Data":"9914fc397eb881303b98cd6db0ec68513c5b85f3bc44f870764a97af4522b967"}
Oct 31 04:04:50.488915 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.488777    4900 scope.go:122] "RemoveContainer" containerID="41d3e3e464977f3aec61abab5a99417f732da6e032c6e6758683e7b1fd6e2e3a"
Oct 31 04:04:50.490232 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.490208    4900 generic.go:356] "Generic (PLEG): container finished" podID="5420545a-5a77-4a15-bea3-8274d49e5e5d" containerID="da24f71e3cc8c08bf687036e910e8c0fcd308e654ae3c71876b0cd52dc3e1216" exitCode=0
Oct 31 04:04:50.490358 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.490249    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x" event={"ID":"5420545a-5a77-4a15-bea3-8274d49e5e5d","Type":"ContainerDied","Data":"da24f71e3cc8c08bf687036e910e8c0fcd308e654ae3c71876b0cd52dc3e1216"}
Oct 31 04:04:50.490358 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.490322    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x" event={"ID":"5420545a-5a77-4a15-bea3-8274d49e5e5d","Type":"ContainerDied","Data":"b917888cabab809d24e8825a0dc44936cecbd3ec350b410ab9525cbc14d8f849"}
Oct 31 04:04:50.490358 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.490253    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"
Oct 31 04:04:50.499115 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.499088    4900 scope.go:122] "RemoveContainer" containerID="41d3e3e464977f3aec61abab5a99417f732da6e032c6e6758683e7b1fd6e2e3a"
Oct 31 04:04:50.499470 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:04:50.499449    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d3e3e464977f3aec61abab5a99417f732da6e032c6e6758683e7b1fd6e2e3a\": container with ID starting with 41d3e3e464977f3aec61abab5a99417f732da6e032c6e6758683e7b1fd6e2e3a not found: ID does not exist" containerID="41d3e3e464977f3aec61abab5a99417f732da6e032c6e6758683e7b1fd6e2e3a"
Oct 31 04:04:50.499577 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.499473    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d3e3e464977f3aec61abab5a99417f732da6e032c6e6758683e7b1fd6e2e3a"} err="failed to get container status \"41d3e3e464977f3aec61abab5a99417f732da6e032c6e6758683e7b1fd6e2e3a\": rpc error: code = NotFound desc = could not find container \"41d3e3e464977f3aec61abab5a99417f732da6e032c6e6758683e7b1fd6e2e3a\": container with ID starting with 41d3e3e464977f3aec61abab5a99417f732da6e032c6e6758683e7b1fd6e2e3a not found: ID does not exist"
Oct 31 04:04:50.499577 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.499489    4900 scope.go:122] "RemoveContainer" containerID="da24f71e3cc8c08bf687036e910e8c0fcd308e654ae3c71876b0cd52dc3e1216"
Oct 31 04:04:50.508572 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.508556    4900 scope.go:122] "RemoveContainer" containerID="da24f71e3cc8c08bf687036e910e8c0fcd308e654ae3c71876b0cd52dc3e1216"
Oct 31 04:04:50.508837 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:04:50.508814    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da24f71e3cc8c08bf687036e910e8c0fcd308e654ae3c71876b0cd52dc3e1216\": container with ID starting with da24f71e3cc8c08bf687036e910e8c0fcd308e654ae3c71876b0cd52dc3e1216 not found: ID does not exist" containerID="da24f71e3cc8c08bf687036e910e8c0fcd308e654ae3c71876b0cd52dc3e1216"
Oct 31 04:04:50.508911 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.508848    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da24f71e3cc8c08bf687036e910e8c0fcd308e654ae3c71876b0cd52dc3e1216"} err="failed to get container status \"da24f71e3cc8c08bf687036e910e8c0fcd308e654ae3c71876b0cd52dc3e1216\": rpc error: code = NotFound desc = could not find container \"da24f71e3cc8c08bf687036e910e8c0fcd308e654ae3c71876b0cd52dc3e1216\": container with ID starting with da24f71e3cc8c08bf687036e910e8c0fcd308e654ae3c71876b0cd52dc3e1216 not found: ID does not exist"
Oct 31 04:04:50.662211 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.662175    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"]
Oct 31 04:04:50.662427 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.662244    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-856ddc-f64f5dddf-wbt7c"]
Oct 31 04:04:50.764708 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.764616    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"]
Oct 31 04:04:50.767264 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:50.767241    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-856ddc-7ddfb87755-7qg7x"]
Oct 31 04:04:51.154614 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:51.154531    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="319c12fa-ccdd-4dac-83e5-802273cc62bf" path="/var/lib/kubelet/pods/319c12fa-ccdd-4dac-83e5-802273cc62bf/volumes"
Oct 31 04:04:51.155125 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:51.155109    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5420545a-5a77-4a15-bea3-8274d49e5e5d" path="/var/lib/kubelet/pods/5420545a-5a77-4a15-bea3-8274d49e5e5d/volumes"
Oct 31 04:04:55.087914 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:55.087875    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:55.569744 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:55.569707    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:04:59.534986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.534950    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"]
Oct 31 04:04:59.535571 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.535557    4900 cpu_manager.go:414] "containerMap: removing container" podUID="319c12fa-ccdd-4dac-83e5-802273cc62bf" containerName="payload-processing"
Oct 31 04:04:59.535612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.535572    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319c12fa-ccdd-4dac-83e5-802273cc62bf" containerName="payload-processing"
Oct 31 04:04:59.535612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.535583    4900 cpu_manager.go:414] "containerMap: removing container" podUID="5420545a-5a77-4a15-bea3-8274d49e5e5d" containerName="payload-pre-processing"
Oct 31 04:04:59.535612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.535588    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5420545a-5a77-4a15-bea3-8274d49e5e5d" containerName="payload-pre-processing"
Oct 31 04:04:59.535714 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.535671    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="319c12fa-ccdd-4dac-83e5-802273cc62bf" containerName="payload-processing"
Oct 31 04:04:59.535714 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.535682    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="5420545a-5a77-4a15-bea3-8274d49e5e5d" containerName="payload-pre-processing"
Oct 31 04:04:59.540432 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.540414    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:04:59.592602 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.592566    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"]
Oct 31 04:04:59.598889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.598862    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-metrics-certs\") pod \"maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:04:59.599208 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.599010    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blnp7\" (UniqueName: \"kubernetes.io/projected/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-kube-api-access-blnp7\") pod \"maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:04:59.599208 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.599149    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:04:59.599418 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.599292    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:04:59.638899 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.638871    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d8190b-serving-cert\""
Oct 31 04:04:59.700487 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.700454    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blnp7\" (UniqueName: \"kubernetes.io/projected/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-kube-api-access-blnp7\") pod \"maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:04:59.700659 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.700526    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:04:59.700659 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.700607    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:04:59.700805 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.700663    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-metrics-certs\") pod \"maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:04:59.700805 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:04:59.700760    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-serving-cert: secret "maas-api-e2e-extmod-a-d8190b-serving-cert" not found
Oct 31 04:04:59.700945 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:04:59.700868    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-maas-api-tls podName:3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3 nodeName:}" failed. No retries permitted until 2026-10-31 04:05:00.200844756 +0000 UTC m=+3453.167178093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-maas-api-tls") pod "maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc" (UID: "3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3") : secret "maas-api-e2e-extmod-a-d8190b-serving-cert" not found
Oct 31 04:04:59.701277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.701251    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:04:59.703132 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.703111    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-metrics-certs\") pod \"maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:04:59.758260 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.758229    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blnp7\" (UniqueName: \"kubernetes.io/projected/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-kube-api-access-blnp7\") pod \"maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:04:59.819247 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:04:59.819167    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:00.029888 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:00.029853    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:00.206239 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:00.206143    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:05:00.208544 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:00.208516    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:05:00.453930 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:00.453895    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:05:00.822258 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:05:00.822227    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5aebad_ba5b_41e0_9df3_b6b4bdae08b3.slice/crio-0839756a2206b627673a78071df6928d47f3a46dd365c41bdc0fbce55d870abf WatchSource:0}: Error finding container 0839756a2206b627673a78071df6928d47f3a46dd365c41bdc0fbce55d870abf: Status 404 returned error can't find the container with id 0839756a2206b627673a78071df6928d47f3a46dd365c41bdc0fbce55d870abf
Oct 31 04:05:00.825296 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:00.825274    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"]
Oct 31 04:05:01.559127 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:01.559093    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc" event={"ID":"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3","Type":"ContainerStarted","Data":"0839756a2206b627673a78071df6928d47f3a46dd365c41bdc0fbce55d870abf"}
Oct 31 04:05:01.730167 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:01.730128    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:02.567167 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:02.567133    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc" event={"ID":"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3","Type":"ContainerStarted","Data":"3b3a007c5bec61e85fdd25a53911e67f85b7b2b8c9ca56292ef02f683f23617a"}
Oct 31 04:05:02.567574 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:02.567324    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:05:02.682684 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:02.682624    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc" podStartSLOduration=2.722969847 podStartE2EDuration="3.682606305s" totalImagesPullingTime="959.636458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:02.682580788 +0000 UTC m=+3455.648914131" watchObservedRunningTime="2026-10-31 04:05:02.682606305 +0000 UTC m=+3455.648939650"
Oct 31 04:05:04.088632 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.088598    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"]
Oct 31 04:05:04.089101 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.088818    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c" podUID="e1f00e6e-3938-47ae-acc4-9a37950c7056" containerName="maas-api" containerID="cri-o://eb36f405b2c3805140e0cc3b4e41571236a7461c6c8ae190287ddb6a1bd2023d" gracePeriod=30
Oct 31 04:05:04.274833 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.274743    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:04.356666 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.356596    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:05:04.457944 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.457908    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-metrics-certs\") pod \"e1f00e6e-3938-47ae-acc4-9a37950c7056\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") "
Oct 31 04:05:04.458119 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.457959    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f00e6e-3938-47ae-acc4-9a37950c7056-kube-api-access-lggxs\" (UniqueName: \"kubernetes.io/projected/e1f00e6e-3938-47ae-acc4-9a37950c7056-kube-api-access-lggxs\") pod \"e1f00e6e-3938-47ae-acc4-9a37950c7056\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") "
Oct 31 04:05:04.458119 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.458017    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-maas-api-tls\") pod \"e1f00e6e-3938-47ae-acc4-9a37950c7056\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") "
Oct 31 04:05:04.458119 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.458098    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1f00e6e-3938-47ae-acc4-9a37950c7056-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f00e6e-3938-47ae-acc4-9a37950c7056-openshift-service-ca\") pod \"e1f00e6e-3938-47ae-acc4-9a37950c7056\" (UID: \"e1f00e6e-3938-47ae-acc4-9a37950c7056\") "
Oct 31 04:05:04.458552 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.458523    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f00e6e-3938-47ae-acc4-9a37950c7056-openshift-service-ca" pod "e1f00e6e-3938-47ae-acc4-9a37950c7056" (UID: "e1f00e6e-3938-47ae-acc4-9a37950c7056"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:04.460207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.460183    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-metrics-certs" pod "e1f00e6e-3938-47ae-acc4-9a37950c7056" (UID: "e1f00e6e-3938-47ae-acc4-9a37950c7056"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:04.460291 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.460201    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-maas-api-tls" pod "e1f00e6e-3938-47ae-acc4-9a37950c7056" (UID: "e1f00e6e-3938-47ae-acc4-9a37950c7056"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:04.460291 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.460284    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f00e6e-3938-47ae-acc4-9a37950c7056-kube-api-access-lggxs" pod "e1f00e6e-3938-47ae-acc4-9a37950c7056" (UID: "e1f00e6e-3938-47ae-acc4-9a37950c7056"). InnerVolumeSpecName "kube-api-access-lggxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:05:04.560025 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.559982    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:04.560025 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.560017    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f00e6e-3938-47ae-acc4-9a37950c7056-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:04.560025 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.560028    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1f00e6e-3938-47ae-acc4-9a37950c7056-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:04.560275 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.560037    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lggxs\" (UniqueName: \"kubernetes.io/projected/e1f00e6e-3938-47ae-acc4-9a37950c7056-kube-api-access-lggxs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:04.581111 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.581075    4900 generic.go:356] "Generic (PLEG): container finished" podID="e1f00e6e-3938-47ae-acc4-9a37950c7056" containerID="eb36f405b2c3805140e0cc3b4e41571236a7461c6c8ae190287ddb6a1bd2023d" exitCode=0
Oct 31 04:05:04.581308 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.581126    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"
Oct 31 04:05:04.581308 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.581162    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c" event={"ID":"e1f00e6e-3938-47ae-acc4-9a37950c7056","Type":"ContainerDied","Data":"eb36f405b2c3805140e0cc3b4e41571236a7461c6c8ae190287ddb6a1bd2023d"}
Oct 31 04:05:04.581308 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.581216    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c" event={"ID":"e1f00e6e-3938-47ae-acc4-9a37950c7056","Type":"ContainerDied","Data":"0506f03e47732df87f27e786740bb43e060b45d5053e0e20049e18c7e9f34c61"}
Oct 31 04:05:04.581308 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.581240    4900 scope.go:122] "RemoveContainer" containerID="eb36f405b2c3805140e0cc3b4e41571236a7461c6c8ae190287ddb6a1bd2023d"
Oct 31 04:05:04.591833 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.591774    4900 scope.go:122] "RemoveContainer" containerID="eb36f405b2c3805140e0cc3b4e41571236a7461c6c8ae190287ddb6a1bd2023d"
Oct 31 04:05:04.592147 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:05:04.592127    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb36f405b2c3805140e0cc3b4e41571236a7461c6c8ae190287ddb6a1bd2023d\": container with ID starting with eb36f405b2c3805140e0cc3b4e41571236a7461c6c8ae190287ddb6a1bd2023d not found: ID does not exist" containerID="eb36f405b2c3805140e0cc3b4e41571236a7461c6c8ae190287ddb6a1bd2023d"
Oct 31 04:05:04.592208 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.592154    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb36f405b2c3805140e0cc3b4e41571236a7461c6c8ae190287ddb6a1bd2023d"} err="failed to get container status \"eb36f405b2c3805140e0cc3b4e41571236a7461c6c8ae190287ddb6a1bd2023d\": rpc error: code = NotFound desc = could not find container \"eb36f405b2c3805140e0cc3b4e41571236a7461c6c8ae190287ddb6a1bd2023d\": container with ID starting with eb36f405b2c3805140e0cc3b4e41571236a7461c6c8ae190287ddb6a1bd2023d not found: ID does not exist"
Oct 31 04:05:04.744984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.744953    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"]
Oct 31 04:05:04.747404 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:04.747385    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-856ddc-6865b96887-srp9c"]
Oct 31 04:05:05.153863 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:05.153763    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f00e6e-3938-47ae-acc4-9a37950c7056" path="/var/lib/kubelet/pods/e1f00e6e-3938-47ae-acc4-9a37950c7056/volumes"
Oct 31 04:05:06.056214 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:06.056176    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:06.987133 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:05:06.987099    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:05:08.579411 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:08.579382    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:05:10.946963 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:10.946932    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:13.293621 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:13.293593    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:13.514677 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:13.514641    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:14.976429 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:14.976392    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:15.441058 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:15.441024    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"]
Oct 31 04:05:15.441311 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:15.441256    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerName="istio-proxy" containerID="cri-o://0a910cf45bf32f35bde6162a6c65a2c0ee7d18a16e5bd318481db7136ca2c1cc" gracePeriod=30
Oct 31 04:05:15.525847 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:15.525808    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:16.227392 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.227363    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:16.507841 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.507804    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:16.762468 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.762375    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"]
Oct 31 04:05:16.763079 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.763059    4900 cpu_manager.go:414] "containerMap: removing container" podUID="e1f00e6e-3938-47ae-acc4-9a37950c7056" containerName="maas-api"
Oct 31 04:05:16.763079 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.763078    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f00e6e-3938-47ae-acc4-9a37950c7056" containerName="maas-api"
Oct 31 04:05:16.763251 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.763209    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f00e6e-3938-47ae-acc4-9a37950c7056" containerName="maas-api"
Oct 31 04:05:16.768403 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.768378    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:16.846957 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.846927    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-23258c-openshift-default-dockercfg-85lp2\""
Oct 31 04:05:16.874093 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.874055    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"]
Oct 31 04:05:16.893001 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.892956    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:16.908466 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.908432    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549b17d1-1fa6-418f-839f-b6e81361aab5-istiod-ca-cert\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:16.908658 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.908483    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-data\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:16.908658 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.908499    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-token\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:16.908658 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.908636    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46w8m\" (UniqueName: \"kubernetes.io/projected/549b17d1-1fa6-418f-839f-b6e81361aab5-kube-api-access-46w8m\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:16.908849 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.908677    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-podinfo\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:16.908849 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.908712    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-workload-socket\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:16.908934 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.908868    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-workload-certs\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:16.908979 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.908959    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-envoy\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:16.909024 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:16.909008    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-credential-socket\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.010614 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.010583    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-credential-socket\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.010840 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.010669    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549b17d1-1fa6-418f-839f-b6e81361aab5-istiod-ca-cert\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.010840 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.010705    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-data\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.010840 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.010722    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-token\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.010840 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.010754    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46w8m\" (UniqueName: \"kubernetes.io/projected/549b17d1-1fa6-418f-839f-b6e81361aab5-kube-api-access-46w8m\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.010840 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.010779    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-podinfo\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.010840 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.010826    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-workload-socket\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.011194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.010936    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-workload-certs\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.011194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.011021    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-envoy\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.011194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.011131    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-credential-socket\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.011321 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.011225    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-data\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.011556 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.011530    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549b17d1-1fa6-418f-839f-b6e81361aab5-istiod-ca-cert\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.011556 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.011552    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-workload-socket\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.011556 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.011530    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-workload-certs\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.013218 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.013167    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-envoy\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.013350 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.013331    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-podinfo\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.068558 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.068523    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46w8m\" (UniqueName: \"kubernetes.io/projected/549b17d1-1fa6-418f-839f-b6e81361aab5-kube-api-access-46w8m\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.068747 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.068725    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-token\") pod \"e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") " pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.085778 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.085750    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:17.100108 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.100082    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:17.509711 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.509677    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"]
Oct 31 04:05:17.541703 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:05:17.541670    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549b17d1_1fa6_418f_839f_b6e81361aab5.slice/crio-1485f670cda2fd69f73e0825ad6212f6425a05d24083b28449999ab060c95105 WatchSource:0}: Error finding container 1485f670cda2fd69f73e0825ad6212f6425a05d24083b28449999ab060c95105: Status 404 returned error can't find the container with id 1485f670cda2fd69f73e0825ad6212f6425a05d24083b28449999ab060c95105
Oct 31 04:05:17.544240 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.544205    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:05:17.544340 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.544277    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:05:17.544340 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.544305    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:05:17.662608 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.662568    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h" event={"ID":"549b17d1-1fa6-418f-839f-b6e81361aab5","Type":"ContainerStarted","Data":"31d30b58e65e889bf5c837df40b264ef7409ac2e7d462938fb7b728e3b865c82"}
Oct 31 04:05:17.662797 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.662630    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h" event={"ID":"549b17d1-1fa6-418f-839f-b6e81361aab5","Type":"ContainerStarted","Data":"1485f670cda2fd69f73e0825ad6212f6425a05d24083b28449999ab060c95105"}
Oct 31 04:05:17.782175 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:17.782015    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h" podStartSLOduration=1.782000984 podStartE2EDuration="1.782000984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:17.781123134 +0000 UTC m=+3470.747456480" watchObservedRunningTime="2026-10-31 04:05:17.782000984 +0000 UTC m=+3470.748334328"
Oct 31 04:05:18.086508 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:18.086420    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:18.092355 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:18.092327    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:18.244622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:18.244591    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:18.470723 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:18.470634    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:18.668647 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:18.668616    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:18.669658 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:18.669640    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:05:18.692135 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:18.692109    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:19.273199 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:19.273163    4900 patch_prober.go:29] interesting pod/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:05:19.273385 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:19.273219    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:05:19.816027 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:19.815992    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:20.682774 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.682739    4900 generic.go:356] "Generic (PLEG): container finished" podID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerID="0a910cf45bf32f35bde6162a6c65a2c0ee7d18a16e5bd318481db7136ca2c1cc" exitCode=0
Oct 31 04:05:20.682989 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.682820    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" event={"ID":"2323f996-abc1-4f51-9ed6-2e96cde226bc","Type":"ContainerDied","Data":"0a910cf45bf32f35bde6162a6c65a2c0ee7d18a16e5bd318481db7136ca2c1cc"}
Oct 31 04:05:20.682989 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.682886    4900 scope.go:122] "RemoveContainer" containerID="a2549ed5802638c525b9c572795964804d4acd67036fefe848f7920d34368a45"
Oct 31 04:05:20.723062 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.723037    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 04:05:20.759731 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.759702    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-envoy\") pod \"2323f996-abc1-4f51-9ed6-2e96cde226bc\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") "
Oct 31 04:05:20.759903 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.759746    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-workload-certs\") pod \"2323f996-abc1-4f51-9ed6-2e96cde226bc\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") "
Oct 31 04:05:20.759903 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.759859    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-workload-socket\") pod \"2323f996-abc1-4f51-9ed6-2e96cde226bc\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") "
Oct 31 04:05:20.760046 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.759932    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2323f996-abc1-4f51-9ed6-2e96cde226bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2323f996-abc1-4f51-9ed6-2e96cde226bc-istiod-ca-cert\") pod \"2323f996-abc1-4f51-9ed6-2e96cde226bc\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") "
Oct 31 04:05:20.760046 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.759970    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-podinfo\") pod \"2323f996-abc1-4f51-9ed6-2e96cde226bc\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") "
Oct 31 04:05:20.760046 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.760018    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2323f996-abc1-4f51-9ed6-2e96cde226bc-kube-api-access-s6gxw\" (UniqueName: \"kubernetes.io/projected/2323f996-abc1-4f51-9ed6-2e96cde226bc-kube-api-access-s6gxw\") pod \"2323f996-abc1-4f51-9ed6-2e96cde226bc\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") "
Oct 31 04:05:20.760046 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.760024    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-workload-certs" pod "2323f996-abc1-4f51-9ed6-2e96cde226bc" (UID: "2323f996-abc1-4f51-9ed6-2e96cde226bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:05:20.760267 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.760125    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-token\" (UniqueName: \"kubernetes.io/projected/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-token\") pod \"2323f996-abc1-4f51-9ed6-2e96cde226bc\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") "
Oct 31 04:05:20.760267 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.760154    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-workload-socket" pod "2323f996-abc1-4f51-9ed6-2e96cde226bc" (UID: "2323f996-abc1-4f51-9ed6-2e96cde226bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:05:20.760267 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.760233    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-credential-socket\") pod \"2323f996-abc1-4f51-9ed6-2e96cde226bc\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") "
Oct 31 04:05:20.760432 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.760269    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-data\") pod \"2323f996-abc1-4f51-9ed6-2e96cde226bc\" (UID: \"2323f996-abc1-4f51-9ed6-2e96cde226bc\") "
Oct 31 04:05:20.760432 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.760340    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2323f996-abc1-4f51-9ed6-2e96cde226bc-istiod-ca-cert" pod "2323f996-abc1-4f51-9ed6-2e96cde226bc" (UID: "2323f996-abc1-4f51-9ed6-2e96cde226bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:20.760543 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.760495    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-credential-socket" pod "2323f996-abc1-4f51-9ed6-2e96cde226bc" (UID: "2323f996-abc1-4f51-9ed6-2e96cde226bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:05:20.760754 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.760723    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-data" pod "2323f996-abc1-4f51-9ed6-2e96cde226bc" (UID: "2323f996-abc1-4f51-9ed6-2e96cde226bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:05:20.761102 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.761070    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:20.761102 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.761097    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:20.761290 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.761125    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2323f996-abc1-4f51-9ed6-2e96cde226bc-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:20.761290 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.761143    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:20.761290 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.761158    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:20.762226 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.762200    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-envoy" pod "2323f996-abc1-4f51-9ed6-2e96cde226bc" (UID: "2323f996-abc1-4f51-9ed6-2e96cde226bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:05:20.762325 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.762285    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-podinfo" pod "2323f996-abc1-4f51-9ed6-2e96cde226bc" (UID: "2323f996-abc1-4f51-9ed6-2e96cde226bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:05:20.762616 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.762601    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2323f996-abc1-4f51-9ed6-2e96cde226bc-kube-api-access-s6gxw" pod "2323f996-abc1-4f51-9ed6-2e96cde226bc" (UID: "2323f996-abc1-4f51-9ed6-2e96cde226bc"). InnerVolumeSpecName "kube-api-access-s6gxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:05:20.762679 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.762623    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-token" pod "2323f996-abc1-4f51-9ed6-2e96cde226bc" (UID: "2323f996-abc1-4f51-9ed6-2e96cde226bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:05:20.862378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.862340    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:20.862378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.862369    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:20.862378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.862382    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6gxw\" (UniqueName: \"kubernetes.io/projected/2323f996-abc1-4f51-9ed6-2e96cde226bc-kube-api-access-s6gxw\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:20.862877 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:20.862393    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2323f996-abc1-4f51-9ed6-2e96cde226bc-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:21.226661 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.226620    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:21.708819 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.692533    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w" event={"ID":"2323f996-abc1-4f51-9ed6-2e96cde226bc","Type":"ContainerDied","Data":"a51ea4d4bb8d5e8383fa7609922d0530c7c32deffe5e81106f2bffaf71b6b4ac"}
Oct 31 04:05:21.708819 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.692574    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"
Oct 31 04:05:21.708819 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.692715    4900 scope.go:122] "RemoveContainer" containerID="0a910cf45bf32f35bde6162a6c65a2c0ee7d18a16e5bd318481db7136ca2c1cc"
Oct 31 04:05:21.865030 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.864969    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"]
Oct 31 04:05:21.865767 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.865745    4900 cpu_manager.go:414] "containerMap: removing container" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerName="istio-proxy"
Oct 31 04:05:21.865767 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.865767    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerName="istio-proxy"
Oct 31 04:05:21.865950 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.865814    4900 cpu_manager.go:414] "containerMap: removing container" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerName="istio-proxy"
Oct 31 04:05:21.865950 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.865823    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerName="istio-proxy"
Oct 31 04:05:21.865950 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.865933    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerName="istio-proxy"
Oct 31 04:05:21.871095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.871076    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:21.930602 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.930567    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"]
Oct 31 04:05:21.933458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.933424    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"]
Oct 31 04:05:21.936113 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.936090    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-856ddc-openshift-default-75fb57976f-7fl4w"]
Oct 31 04:05:21.957429 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.957403    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-0d5219-openshift-default-dockercfg-qh68f\""
Oct 31 04:05:21.974497 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.974424    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-envoy\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:21.974497 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.974473    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-podinfo\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:21.974675 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.974601    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-data\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:21.974675 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.974661    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-credential-socket\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:21.974825 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.974690    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-workload-certs\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:21.974825 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.974723    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-workload-socket\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:21.974955 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.974859    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqjxm\" (UniqueName: \"kubernetes.io/projected/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-kube-api-access-qqjxm\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:21.975019 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.974980    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-token\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:21.975077 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:21.975018    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istiod-ca-cert\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.007204 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.007167    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:22.076717 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.076681    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-token\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.076717 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.076718    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istiod-ca-cert\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.076990 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.076749    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-envoy\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.076990 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.076800    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-podinfo\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.076990 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.076848    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-data\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.076990 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.076881    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-credential-socket\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.076990 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.076907    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-workload-certs\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.076990 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.076962    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-workload-socket\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.077317 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.077000    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqjxm\" (UniqueName: \"kubernetes.io/projected/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-kube-api-access-qqjxm\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.077317 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.077272    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-data\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.077398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.077367    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-credential-socket\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.077446 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.077396    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-workload-certs\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.077603 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.077581    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istiod-ca-cert\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.077643 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.077613    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-workload-socket\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.079130 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.079110    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-envoy\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.079354 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.079338    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-podinfo\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.133422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.133388    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-token\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.133573 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.133555    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqjxm\" (UniqueName: \"kubernetes.io/projected/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-kube-api-access-qqjxm\") pod \"e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") " pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.185976 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.185943    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:22.229456 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.229354    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:22.444228 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.444192    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:22.606030 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:05:22.605994    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1a6b6b_c18f_40f6_b408_0090cafe29e1.slice/crio-d1ee3be64e1325efc77ca17989c81e0b5b469c2922c96b13a4f97c43269a4f49 WatchSource:0}: Error finding container d1ee3be64e1325efc77ca17989c81e0b5b469c2922c96b13a4f97c43269a4f49: Status 404 returned error can't find the container with id d1ee3be64e1325efc77ca17989c81e0b5b469c2922c96b13a4f97c43269a4f49
Oct 31 04:05:22.608671 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.608628    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:05:22.608782 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.608706    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:05:22.608782 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.608737    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:05:22.609483 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.609463    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"]
Oct 31 04:05:22.702120 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.702085    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p" event={"ID":"bc1a6b6b-c18f-40f6-b408-0090cafe29e1","Type":"ContainerStarted","Data":"26113adcfa27137b1c1974bbea82b46e306b09b5e0cc6d6a556d1f38123e37f2"}
Oct 31 04:05:22.702256 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.702143    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p" event={"ID":"bc1a6b6b-c18f-40f6-b408-0090cafe29e1","Type":"ContainerStarted","Data":"d1ee3be64e1325efc77ca17989c81e0b5b469c2922c96b13a4f97c43269a4f49"}
Oct 31 04:05:22.822657 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:22.822605    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p" podStartSLOduration=1.8225901389999999 podStartE2EDuration="1.822590139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:22.819647232 +0000 UTC m=+3475.785980575" watchObservedRunningTime="2026-10-31 04:05:22.822590139 +0000 UTC m=+3475.788923487"
Oct 31 04:05:23.155098 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:23.155066    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" path="/var/lib/kubelet/pods/2323f996-abc1-4f51-9ed6-2e96cde226bc/volumes"
Oct 31 04:05:23.186313 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:23.186287    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:23.191954 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:23.191932    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:23.708487 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:23.708458    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:23.709319 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:23.709299    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:05:24.115447 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:24.115420    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:25.752590 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:25.752550    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:25.971993 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:25.971924    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:26.189976 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:26.189898    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:27.530447 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:27.530414    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:29.106258 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:29.106225    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:29.299401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:29.299370    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:29.779257 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:29.779225    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"]
Oct 31 04:05:29.779976 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:29.779959    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="2323f996-abc1-4f51-9ed6-2e96cde226bc" containerName="istio-proxy"
Oct 31 04:05:29.784882 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:29.784866    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"
Oct 31 04:05:29.835270 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:29.835248    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"]
Oct 31 04:05:29.874834 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:29.874777    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs8xf\" (UniqueName: \"kubernetes.io/projected/0280841e-0df8-4917-bde3-e88f014d1151-kube-api-access-bs8xf\") pod \"payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg\" (UID: \"0280841e-0df8-4917-bde3-e88f014d1151\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"
Oct 31 04:05:29.874970 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:29.874839    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0280841e-0df8-4917-bde3-e88f014d1151-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg\" (UID: \"0280841e-0df8-4917-bde3-e88f014d1151\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"
Oct 31 04:05:29.882167 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:29.882151    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0d5219\""
Oct 31 04:05:29.882167 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:29.882161    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0d5219-dockercfg-bvf8g\""
Oct 31 04:05:29.976276 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:29.976246    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs8xf\" (UniqueName: \"kubernetes.io/projected/0280841e-0df8-4917-bde3-e88f014d1151-kube-api-access-bs8xf\") pod \"payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg\" (UID: \"0280841e-0df8-4917-bde3-e88f014d1151\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"
Oct 31 04:05:29.976430 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:29.976280    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0280841e-0df8-4917-bde3-e88f014d1151-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg\" (UID: \"0280841e-0df8-4917-bde3-e88f014d1151\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"
Oct 31 04:05:29.976947 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:29.976929    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0280841e-0df8-4917-bde3-e88f014d1151-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg\" (UID: \"0280841e-0df8-4917-bde3-e88f014d1151\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"
Oct 31 04:05:30.032622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:30.032550    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs8xf\" (UniqueName: \"kubernetes.io/projected/0280841e-0df8-4917-bde3-e88f014d1151-kube-api-access-bs8xf\") pod \"payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg\" (UID: \"0280841e-0df8-4917-bde3-e88f014d1151\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"
Oct 31 04:05:30.098872 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:30.098847    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"
Oct 31 04:05:30.468311 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:05:30.468280    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0280841e_0df8_4917_bde3_e88f014d1151.slice/crio-b16693dc065acdf0017bd6a84c3bc7b793a2b1cfe64647a80e98b4459a5f388b WatchSource:0}: Error finding container b16693dc065acdf0017bd6a84c3bc7b793a2b1cfe64647a80e98b4459a5f388b: Status 404 returned error can't find the container with id b16693dc065acdf0017bd6a84c3bc7b793a2b1cfe64647a80e98b4459a5f388b
Oct 31 04:05:30.470667 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:30.470645    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"]
Oct 31 04:05:30.750556 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:30.750525    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg" event={"ID":"0280841e-0df8-4917-bde3-e88f014d1151","Type":"ContainerStarted","Data":"9606c3c6f9e6bc45e7ca24502decc845776f724eedb9601c81d9c58ec70f4645"}
Oct 31 04:05:30.750733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:30.750582    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"
Oct 31 04:05:30.750733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:30.750599    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg" event={"ID":"0280841e-0df8-4917-bde3-e88f014d1151","Type":"ContainerStarted","Data":"b16693dc065acdf0017bd6a84c3bc7b793a2b1cfe64647a80e98b4459a5f388b"}
Oct 31 04:05:30.867386 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:30.867136    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg" podStartSLOduration=1.8671187580000002 podStartE2EDuration="1.867118758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:05:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:30.864977762 +0000 UTC m=+3483.831311106" watchObservedRunningTime="2026-10-31 04:05:30.867118758 +0000 UTC m=+3483.833452104"
Oct 31 04:05:32.476950 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:32.476915    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:36.984636 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:05:36.984598    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:05:37.397083 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:37.396998    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:41.761043 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:41.761011    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"
Oct 31 04:05:42.310051 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:42.310014    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:47.081336 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:47.081306    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:47.291403 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:47.291368    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:51.994509 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:51.994476    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:56.698922 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:56.698482    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:56.905974 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:56.905943    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:57.050093 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:57.050059    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"]
Oct 31 04:05:57.099456 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:57.099424    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61784e4f-b7zv8"]
Oct 31 04:05:57.108102 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:57.108050    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:05:57.153444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:05:57.153406    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf79a1c1-06f1-4f9d-bd60-de74abfaf581" path="/var/lib/kubelet/pods/bf79a1c1-06f1-4f9d-bd60-de74abfaf581/volumes"
Oct 31 04:06:00.973825 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:00.973780    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"]
Oct 31 04:06:00.974283 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:00.973992    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h" podUID="549b17d1-1fa6-418f-839f-b6e81361aab5" containerName="istio-proxy" containerID="cri-o://31d30b58e65e889bf5c837df40b264ef7409ac2e7d462938fb7b728e3b865c82" gracePeriod=30
Oct 31 04:06:01.045138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:01.045096    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:01.412198 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:01.412147    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:01.613278 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:01.613212    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:01.985478 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:01.985439    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:02.192386 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:02.192355    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:03.671135 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:03.671095    4900 patch_prober.go:29] interesting pod/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:06:03.671534 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:03.671177    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h" podUID="549b17d1-1fa6-418f-839f-b6e81361aab5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:06:06.235468 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.235439    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:06:06.322971 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.322939    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549b17d1-1fa6-418f-839f-b6e81361aab5-kube-api-access-46w8m\" (UniqueName: \"kubernetes.io/projected/549b17d1-1fa6-418f-839f-b6e81361aab5-kube-api-access-46w8m\") pod \"549b17d1-1fa6-418f-839f-b6e81361aab5\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") "
Oct 31 04:06:06.323138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.323031    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-token\" (UniqueName: \"kubernetes.io/projected/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-token\") pod \"549b17d1-1fa6-418f-839f-b6e81361aab5\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") "
Oct 31 04:06:06.323138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.323090    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-credential-socket\") pod \"549b17d1-1fa6-418f-839f-b6e81361aab5\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") "
Oct 31 04:06:06.323231 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.323178    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-workload-socket\") pod \"549b17d1-1fa6-418f-839f-b6e81361aab5\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") "
Oct 31 04:06:06.323269 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.323226    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-podinfo\") pod \"549b17d1-1fa6-418f-839f-b6e81361aab5\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") "
Oct 31 04:06:06.323323 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.323283    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-envoy\") pod \"549b17d1-1fa6-418f-839f-b6e81361aab5\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") "
Oct 31 04:06:06.323396 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.323376    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-data\") pod \"549b17d1-1fa6-418f-839f-b6e81361aab5\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") "
Oct 31 04:06:06.323536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.323493    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/549b17d1-1fa6-418f-839f-b6e81361aab5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549b17d1-1fa6-418f-839f-b6e81361aab5-istiod-ca-cert\") pod \"549b17d1-1fa6-418f-839f-b6e81361aab5\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") "
Oct 31 04:06:06.323536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.323396    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-credential-socket" pod "549b17d1-1fa6-418f-839f-b6e81361aab5" (UID: "549b17d1-1fa6-418f-839f-b6e81361aab5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:06:06.323536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.323417    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-workload-socket" pod "549b17d1-1fa6-418f-839f-b6e81361aab5" (UID: "549b17d1-1fa6-418f-839f-b6e81361aab5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:06:06.323707 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.323535    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-workload-certs\") pod \"549b17d1-1fa6-418f-839f-b6e81361aab5\" (UID: \"549b17d1-1fa6-418f-839f-b6e81361aab5\") "
Oct 31 04:06:06.323707 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.323638    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-data" pod "549b17d1-1fa6-418f-839f-b6e81361aab5" (UID: "549b17d1-1fa6-418f-839f-b6e81361aab5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:06:06.323985 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.323953    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-workload-certs" pod "549b17d1-1fa6-418f-839f-b6e81361aab5" (UID: "549b17d1-1fa6-418f-839f-b6e81361aab5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:06:06.323985 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.323876    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549b17d1-1fa6-418f-839f-b6e81361aab5-istiod-ca-cert" pod "549b17d1-1fa6-418f-839f-b6e81361aab5" (UID: "549b17d1-1fa6-418f-839f-b6e81361aab5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:06:06.324324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.324304    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:06.324408 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.324329    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:06.324408 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.324344    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:06.324408 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.324358    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/549b17d1-1fa6-418f-839f-b6e81361aab5-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:06.324408 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.324372    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:06.325289 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.325266    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-token" pod "549b17d1-1fa6-418f-839f-b6e81361aab5" (UID: "549b17d1-1fa6-418f-839f-b6e81361aab5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:06:06.325615 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.325589    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549b17d1-1fa6-418f-839f-b6e81361aab5-kube-api-access-46w8m" pod "549b17d1-1fa6-418f-839f-b6e81361aab5" (UID: "549b17d1-1fa6-418f-839f-b6e81361aab5"). InnerVolumeSpecName "kube-api-access-46w8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:06:06.325615 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.325590    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-envoy" pod "549b17d1-1fa6-418f-839f-b6e81361aab5" (UID: "549b17d1-1fa6-418f-839f-b6e81361aab5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:06:06.325740 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.325658    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-podinfo" pod "549b17d1-1fa6-418f-839f-b6e81361aab5" (UID: "549b17d1-1fa6-418f-839f-b6e81361aab5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:06:06.425994 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.425959    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46w8m\" (UniqueName: \"kubernetes.io/projected/549b17d1-1fa6-418f-839f-b6e81361aab5-kube-api-access-46w8m\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:06.425994 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.425998    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:06.426182 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.426014    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:06.426182 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.426029    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/549b17d1-1fa6-418f-839f-b6e81361aab5-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:06.585837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.585720    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:06.876199 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.876077    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:06.978255 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.978158    4900 generic.go:356] "Generic (PLEG): container finished" podID="549b17d1-1fa6-418f-839f-b6e81361aab5" containerID="31d30b58e65e889bf5c837df40b264ef7409ac2e7d462938fb7b728e3b865c82" exitCode=0
Oct 31 04:06:06.978255 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.978221    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"
Oct 31 04:06:06.978504 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.978250    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h" event={"ID":"549b17d1-1fa6-418f-839f-b6e81361aab5","Type":"ContainerDied","Data":"31d30b58e65e889bf5c837df40b264ef7409ac2e7d462938fb7b728e3b865c82"}
Oct 31 04:06:06.978504 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.978334    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h" event={"ID":"549b17d1-1fa6-418f-839f-b6e81361aab5","Type":"ContainerDied","Data":"1485f670cda2fd69f73e0825ad6212f6425a05d24083b28449999ab060c95105"}
Oct 31 04:06:06.978504 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.978374    4900 scope.go:122] "RemoveContainer" containerID="31d30b58e65e889bf5c837df40b264ef7409ac2e7d462938fb7b728e3b865c82"
Oct 31 04:06:06.986727 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:06:06.986696    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:06:06.990732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.990712    4900 scope.go:122] "RemoveContainer" containerID="31d30b58e65e889bf5c837df40b264ef7409ac2e7d462938fb7b728e3b865c82"
Oct 31 04:06:06.991007 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:06:06.990989    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31d30b58e65e889bf5c837df40b264ef7409ac2e7d462938fb7b728e3b865c82\": container with ID starting with 31d30b58e65e889bf5c837df40b264ef7409ac2e7d462938fb7b728e3b865c82 not found: ID does not exist" containerID="31d30b58e65e889bf5c837df40b264ef7409ac2e7d462938fb7b728e3b865c82"
Oct 31 04:06:06.991065 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:06.991015    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31d30b58e65e889bf5c837df40b264ef7409ac2e7d462938fb7b728e3b865c82"} err="failed to get container status \"31d30b58e65e889bf5c837df40b264ef7409ac2e7d462938fb7b728e3b865c82\": rpc error: code = NotFound desc = could not find container \"31d30b58e65e889bf5c837df40b264ef7409ac2e7d462938fb7b728e3b865c82\": container with ID starting with 31d30b58e65e889bf5c837df40b264ef7409ac2e7d462938fb7b728e3b865c82 not found: ID does not exist"
Oct 31 04:06:07.193405 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:07.193308    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"]
Oct 31 04:06:07.244880 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:07.244851    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-23258c-openshift-default-6ccf7f9cdc-hxb5h"]
Oct 31 04:06:08.054006 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:08.053970    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"]
Oct 31 04:06:08.092047 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:08.092005    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fdd6806-dz5mg"]
Oct 31 04:06:09.153650 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:09.153571    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb2b71f-e32a-4b82-93f6-b48e3b41ca35" path="/var/lib/kubelet/pods/0cb2b71f-e32a-4b82-93f6-b48e3b41ca35/volumes"
Oct 31 04:06:09.154233 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:09.154216    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549b17d1-1fa6-418f-839f-b6e81361aab5" path="/var/lib/kubelet/pods/549b17d1-1fa6-418f-839f-b6e81361aab5/volumes"
Oct 31 04:06:11.787685 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:11.787650    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:13.579957 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:13.579921    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:13.796152 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:13.796121    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:15.088698 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:15.088658    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:15.544900 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:15.544869    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:16.021835 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.021802    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"]
Oct 31 04:06:16.022397 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.022384    4900 cpu_manager.go:414] "containerMap: removing container" podUID="549b17d1-1fa6-418f-839f-b6e81361aab5" containerName="istio-proxy"
Oct 31 04:06:16.022448 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.022399    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549b17d1-1fa6-418f-839f-b6e81361aab5" containerName="istio-proxy"
Oct 31 04:06:16.022519 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.022509    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="549b17d1-1fa6-418f-839f-b6e81361aab5" containerName="istio-proxy"
Oct 31 04:06:16.025951 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.025936    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.046347 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.046325    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-workload-socket\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.046484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.046370    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0187126e-af1f-44bf-9cef-13217cb07672-istio-podinfo\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.046484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.046448    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6g9j\" (UniqueName: \"kubernetes.io/projected/0187126e-af1f-44bf-9cef-13217cb07672-kube-api-access-l6g9j\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.046579 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.046481    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-istio-data\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.046642 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.046584    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-istio-envoy\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.046642 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.046608    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-workload-certs\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.046763 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.046737    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0187126e-af1f-44bf-9cef-13217cb07672-istiod-ca-cert\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.046846 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.046771    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0187126e-af1f-44bf-9cef-13217cb07672-istio-token\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.046909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.046892    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-credential-socket\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.108533 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.108508    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-cfbb5b-openshift-default-dockercfg-hr69b\""
Oct 31 04:06:16.134678 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.134642    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"]
Oct 31 04:06:16.147977 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.147949    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-credential-socket\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.148138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.148015    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-workload-socket\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.148138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.148067    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0187126e-af1f-44bf-9cef-13217cb07672-istio-podinfo\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.148138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.148111    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6g9j\" (UniqueName: \"kubernetes.io/projected/0187126e-af1f-44bf-9cef-13217cb07672-kube-api-access-l6g9j\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.148324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.148153    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-istio-data\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.148324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.148229    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-istio-envoy\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.148324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.148288    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-workload-certs\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.148483 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.148388    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0187126e-af1f-44bf-9cef-13217cb07672-istiod-ca-cert\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.148483 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.148424    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0187126e-af1f-44bf-9cef-13217cb07672-istio-token\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.148483 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.148458    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-workload-socket\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.148832 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.148810    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-istio-data\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.148931 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.148854    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-workload-certs\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.149245 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.149221    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0187126e-af1f-44bf-9cef-13217cb07672-istiod-ca-cert\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.149484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.149460    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-credential-socket\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.151251 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.151222    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-istio-envoy\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.152091 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.152064    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0187126e-af1f-44bf-9cef-13217cb07672-istio-podinfo\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.222536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.222502    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6g9j\" (UniqueName: \"kubernetes.io/projected/0187126e-af1f-44bf-9cef-13217cb07672-kube-api-access-l6g9j\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.225554 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.225502    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0187126e-af1f-44bf-9cef-13217cb07672-istio-token\") pod \"e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") " pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.300076 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.299989    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:16.340402 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.340375    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:16.507642 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.507614    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:16.719356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.719278    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:16.756857 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:06:16.756831    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0187126e_af1f_44bf_9cef_13217cb07672.slice/crio-bedd9fe14fe772958382e8eded601af761a6bc9fc97d862965ada452bf91d0d3 WatchSource:0}: Error finding container bedd9fe14fe772958382e8eded601af761a6bc9fc97d862965ada452bf91d0d3: Status 404 returned error can't find the container with id bedd9fe14fe772958382e8eded601af761a6bc9fc97d862965ada452bf91d0d3
Oct 31 04:06:16.759282 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.759251    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:06:16.759371 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.759315    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:06:16.759371 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.759348    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:06:16.759633 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:16.759616    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"]
Oct 31 04:06:17.055399 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:17.055365    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s" event={"ID":"0187126e-af1f-44bf-9cef-13217cb07672","Type":"ContainerStarted","Data":"69f01b93e95d8198bc82087298eff196179c18dba7f92ebc9d4d6904de6b10aa"}
Oct 31 04:06:17.055616 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:17.055449    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s" event={"ID":"0187126e-af1f-44bf-9cef-13217cb07672","Type":"ContainerStarted","Data":"bedd9fe14fe772958382e8eded601af761a6bc9fc97d862965ada452bf91d0d3"}
Oct 31 04:06:17.103309 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:17.103272    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:17.176824 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:17.176723    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s" podStartSLOduration=2.176706526 podStartE2EDuration="2.176706526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:17.17276081 +0000 UTC m=+3530.139094151" watchObservedRunningTime="2026-10-31 04:06:17.176706526 +0000 UTC m=+3530.143039873"
Oct 31 04:06:17.341407 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:17.341308    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:17.347043 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:17.347017    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:18.062490 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:18.062455    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:18.063497 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:18.063478    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:06:20.052780 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:20.052746    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:20.284040 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:20.284005    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:22.608312 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:22.608284    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:23.354964 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:23.354930    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:23.557933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:23.557900    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:23.844938 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:23.844897    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"]
Oct 31 04:06:23.850260 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:23.850234    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"
Oct 31 04:06:23.904834 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:23.900447    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"]
Oct 31 04:06:23.946151 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:23.946122    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-cfbb5b-dockercfg-mgbst\""
Oct 31 04:06:23.946151 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:23.946136    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-cfbb5b\""
Oct 31 04:06:23.953866 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:23.953837    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrxk\" (UniqueName: \"kubernetes.io/projected/2f956311-c81e-4977-bfa9-d1084488abeb-kube-api-access-tjrxk\") pod \"payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk\" (UID: \"2f956311-c81e-4977-bfa9-d1084488abeb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"
Oct 31 04:06:23.953980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:23.953890    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f956311-c81e-4977-bfa9-d1084488abeb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk\" (UID: \"2f956311-c81e-4977-bfa9-d1084488abeb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"
Oct 31 04:06:24.054697 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:24.054664    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrxk\" (UniqueName: \"kubernetes.io/projected/2f956311-c81e-4977-bfa9-d1084488abeb-kube-api-access-tjrxk\") pod \"payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk\" (UID: \"2f956311-c81e-4977-bfa9-d1084488abeb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"
Oct 31 04:06:24.054914 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:24.054717    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f956311-c81e-4977-bfa9-d1084488abeb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk\" (UID: \"2f956311-c81e-4977-bfa9-d1084488abeb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"
Oct 31 04:06:24.055687 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:24.055662    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f956311-c81e-4977-bfa9-d1084488abeb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk\" (UID: \"2f956311-c81e-4977-bfa9-d1084488abeb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"
Oct 31 04:06:24.112461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:24.112384    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrxk\" (UniqueName: \"kubernetes.io/projected/2f956311-c81e-4977-bfa9-d1084488abeb-kube-api-access-tjrxk\") pod \"payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk\" (UID: \"2f956311-c81e-4977-bfa9-d1084488abeb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"
Oct 31 04:06:24.195368 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:24.195334    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"
Oct 31 04:06:24.563374 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:06:24.563342    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f956311_c81e_4977_bfa9_d1084488abeb.slice/crio-dd9c47365d4d043bb06748a7cef9e9ad8e847f954e97c7ff4e09bb9e6e84d6fe WatchSource:0}: Error finding container dd9c47365d4d043bb06748a7cef9e9ad8e847f954e97c7ff4e09bb9e6e84d6fe: Status 404 returned error can't find the container with id dd9c47365d4d043bb06748a7cef9e9ad8e847f954e97c7ff4e09bb9e6e84d6fe
Oct 31 04:06:24.565985 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:24.565961    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"]
Oct 31 04:06:25.108906 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:25.108870    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk" event={"ID":"2f956311-c81e-4977-bfa9-d1084488abeb","Type":"ContainerStarted","Data":"56fdc4bf5e863905abdaf662dbf3df02e71688f4f1600730622022f029bd6d12"}
Oct 31 04:06:25.109345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:25.108925    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk" event={"ID":"2f956311-c81e-4977-bfa9-d1084488abeb","Type":"ContainerStarted","Data":"dd9c47365d4d043bb06748a7cef9e9ad8e847f954e97c7ff4e09bb9e6e84d6fe"}
Oct 31 04:06:25.109345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:25.109021    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"
Oct 31 04:06:25.223946 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:25.223892    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk" podStartSLOduration=2.223876626 podStartE2EDuration="2.223876626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:25.221652119 +0000 UTC m=+3538.187985462" watchObservedRunningTime="2026-10-31 04:06:25.223876626 +0000 UTC m=+3538.190210008"
Oct 31 04:06:28.054277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:28.054233    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:33.460216 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:33.460183    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:36.119233 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:36.119204    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"
Oct 31 04:06:36.984401 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:06:36.984360    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:06:37.989701 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:37.989659    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:38.721922 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:38.721887    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:43.681364 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:43.681331    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:43.893568 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:43.893527    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:44.337989 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:44.337955    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:44.720203 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:44.720115    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"]
Oct 31 04:06:45.056807 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:45.056200    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:45.267216 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:45.267180    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:45.471014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:45.470936    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:45.941957 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:45.941926    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:46.190264 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:46.190214    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:49.073052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:49.073016    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:49.720860 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:49.720821    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g" podUID="2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631" containerName="payload-pre-processing" containerID="cri-o://cd131b06fe3eb32e79d3ebf7d4cf5d9023c141da2f1b5df48e29a35837304fd4" gracePeriod=25
Oct 31 04:06:49.988031 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:49.988005    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"
Oct 31 04:06:49.995352 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:49.995307    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:50.104014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.103982    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631-kube-api-access-52l9b\" (UniqueName: \"kubernetes.io/projected/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631-kube-api-access-52l9b\") pod \"2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631\" (UID: \"2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631\") "
Oct 31 04:06:50.104524 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.104266    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631-plugins-config-volume\") pod \"2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631\" (UID: \"2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631\") "
Oct 31 04:06:50.104642 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.104619    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631-plugins-config-volume" pod "2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631" (UID: "2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:06:50.104799 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.104770    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:50.106733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.106702    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631-kube-api-access-52l9b" pod "2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631" (UID: "2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631"). InnerVolumeSpecName "kube-api-access-52l9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:06:50.206983 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.206943    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52l9b\" (UniqueName: \"kubernetes.io/projected/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631-kube-api-access-52l9b\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:06:50.267541 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.267457    4900 generic.go:356] "Generic (PLEG): container finished" podID="2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631" containerID="cd131b06fe3eb32e79d3ebf7d4cf5d9023c141da2f1b5df48e29a35837304fd4" exitCode=0
Oct 31 04:06:50.267541 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.267509    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"
Oct 31 04:06:50.267728 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.267542    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g" event={"ID":"2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631","Type":"ContainerDied","Data":"cd131b06fe3eb32e79d3ebf7d4cf5d9023c141da2f1b5df48e29a35837304fd4"}
Oct 31 04:06:50.267728 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.267604    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g" event={"ID":"2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631","Type":"ContainerDied","Data":"15b7b2efa5497c61be5baca59944d2d67132c249c26aaddaec803c5d58519c2f"}
Oct 31 04:06:50.267728 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.267631    4900 scope.go:122] "RemoveContainer" containerID="cd131b06fe3eb32e79d3ebf7d4cf5d9023c141da2f1b5df48e29a35837304fd4"
Oct 31 04:06:50.277710 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.277691    4900 scope.go:122] "RemoveContainer" containerID="cd131b06fe3eb32e79d3ebf7d4cf5d9023c141da2f1b5df48e29a35837304fd4"
Oct 31 04:06:50.278005 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:06:50.277986    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd131b06fe3eb32e79d3ebf7d4cf5d9023c141da2f1b5df48e29a35837304fd4\": container with ID starting with cd131b06fe3eb32e79d3ebf7d4cf5d9023c141da2f1b5df48e29a35837304fd4 not found: ID does not exist" containerID="cd131b06fe3eb32e79d3ebf7d4cf5d9023c141da2f1b5df48e29a35837304fd4"
Oct 31 04:06:50.278082 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.278017    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd131b06fe3eb32e79d3ebf7d4cf5d9023c141da2f1b5df48e29a35837304fd4"} err="failed to get container status \"cd131b06fe3eb32e79d3ebf7d4cf5d9023c141da2f1b5df48e29a35837304fd4\": rpc error: code = NotFound desc = could not find container \"cd131b06fe3eb32e79d3ebf7d4cf5d9023c141da2f1b5df48e29a35837304fd4\": container with ID starting with cd131b06fe3eb32e79d3ebf7d4cf5d9023c141da2f1b5df48e29a35837304fd4 not found: ID does not exist"
Oct 31 04:06:50.432295 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.432257    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"]
Oct 31 04:06:50.437553 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.437521    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d8190b-57d559946c-sdx7g"]
Oct 31 04:06:50.736278 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.736250    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:50.957526 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:50.957493    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:51.154052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:51.153962    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631" path="/var/lib/kubelet/pods/2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631/volumes"
Oct 31 04:06:54.028388 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.028351    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:54.245265 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.245235    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:54.369118 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.369037    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"]
Oct 31 04:06:54.369695 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.369678    4900 cpu_manager.go:414] "containerMap: removing container" podUID="2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631" containerName="payload-pre-processing"
Oct 31 04:06:54.369743 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.369698    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631" containerName="payload-pre-processing"
Oct 31 04:06:54.369824 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.369815    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc5ed9f-360f-4bbd-a1f8-3eea73f2b631" containerName="payload-pre-processing"
Oct 31 04:06:54.374733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.374718    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"
Oct 31 04:06:54.426113 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.426079    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"]
Oct 31 04:06:54.446189 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.446161    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:06:54.459149 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.459119    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0050108d-215a-467c-8244-3c882c1b29c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv\" (UID: \"0050108d-215a-467c-8244-3c882c1b29c6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"
Oct 31 04:06:54.459324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.459270    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g529\" (UniqueName: \"kubernetes.io/projected/0050108d-215a-467c-8244-3c882c1b29c6-kube-api-access-8g529\") pod \"payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv\" (UID: \"0050108d-215a-467c-8244-3c882c1b29c6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"
Oct 31 04:06:54.472059 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.472031    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d253a5-dockercfg-2vbwl\""
Oct 31 04:06:54.472290 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.472124    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d253a5\""
Oct 31 04:06:54.518873 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.518839    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"]
Oct 31 04:06:54.524146 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.524118    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"
Oct 31 04:06:54.560529 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.560496    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0050108d-215a-467c-8244-3c882c1b29c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv\" (UID: \"0050108d-215a-467c-8244-3c882c1b29c6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"
Oct 31 04:06:54.560810 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.560586    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g529\" (UniqueName: \"kubernetes.io/projected/0050108d-215a-467c-8244-3c882c1b29c6-kube-api-access-8g529\") pod \"payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv\" (UID: \"0050108d-215a-467c-8244-3c882c1b29c6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"
Oct 31 04:06:54.561358 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.561332    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0050108d-215a-467c-8244-3c882c1b29c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv\" (UID: \"0050108d-215a-467c-8244-3c882c1b29c6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"
Oct 31 04:06:54.577726 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.577694    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"]
Oct 31 04:06:54.617631 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.617597    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g529\" (UniqueName: \"kubernetes.io/projected/0050108d-215a-467c-8244-3c882c1b29c6-kube-api-access-8g529\") pod \"payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv\" (UID: \"0050108d-215a-467c-8244-3c882c1b29c6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"
Oct 31 04:06:54.662255 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.662165    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7190e98-41c5-426a-b7a3-ded4161c3266-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm\" (UID: \"d7190e98-41c5-426a-b7a3-ded4161c3266\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"
Oct 31 04:06:54.662422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.662319    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v66l\" (UniqueName: \"kubernetes.io/projected/d7190e98-41c5-426a-b7a3-ded4161c3266-kube-api-access-8v66l\") pod \"payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm\" (UID: \"d7190e98-41c5-426a-b7a3-ded4161c3266\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"
Oct 31 04:06:54.688487 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.688458    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"
Oct 31 04:06:54.763929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.763844    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7190e98-41c5-426a-b7a3-ded4161c3266-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm\" (UID: \"d7190e98-41c5-426a-b7a3-ded4161c3266\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"
Oct 31 04:06:54.764094 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.764056    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v66l\" (UniqueName: \"kubernetes.io/projected/d7190e98-41c5-426a-b7a3-ded4161c3266-kube-api-access-8v66l\") pod \"payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm\" (UID: \"d7190e98-41c5-426a-b7a3-ded4161c3266\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"
Oct 31 04:06:54.764716 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.764692    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7190e98-41c5-426a-b7a3-ded4161c3266-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm\" (UID: \"d7190e98-41c5-426a-b7a3-ded4161c3266\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"
Oct 31 04:06:54.825420 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.825387    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v66l\" (UniqueName: \"kubernetes.io/projected/d7190e98-41c5-426a-b7a3-ded4161c3266-kube-api-access-8v66l\") pod \"payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm\" (UID: \"d7190e98-41c5-426a-b7a3-ded4161c3266\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"
Oct 31 04:06:54.839288 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:54.839259    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"
Oct 31 04:06:55.058905 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:06:55.058874    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0050108d_215a_467c_8244_3c882c1b29c6.slice/crio-d74bc94df3b3dcb54cebb3afc1eabb1a7c79356200d23972fc966edbf03ffb7f WatchSource:0}: Error finding container d74bc94df3b3dcb54cebb3afc1eabb1a7c79356200d23972fc966edbf03ffb7f: Status 404 returned error can't find the container with id d74bc94df3b3dcb54cebb3afc1eabb1a7c79356200d23972fc966edbf03ffb7f
Oct 31 04:06:55.061199 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:55.061171    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"]
Oct 31 04:06:55.210529 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:06:55.210498    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7190e98_41c5_426a_b7a3_ded4161c3266.slice/crio-b227fd8410a6f97bd863042f557dce24d7ace986b0f7b75f7899049757b5c8cf WatchSource:0}: Error finding container b227fd8410a6f97bd863042f557dce24d7ace986b0f7b75f7899049757b5c8cf: Status 404 returned error can't find the container with id b227fd8410a6f97bd863042f557dce24d7ace986b0f7b75f7899049757b5c8cf
Oct 31 04:06:55.213088 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:55.213062    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"]
Oct 31 04:06:55.303437 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:55.303401    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv" event={"ID":"0050108d-215a-467c-8244-3c882c1b29c6","Type":"ContainerStarted","Data":"d035a395a96cea91cd237af128b06a69d4e3a1c04698f7b4627d0d49e4a4ee11"}
Oct 31 04:06:55.303637 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:55.303468    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv" event={"ID":"0050108d-215a-467c-8244-3c882c1b29c6","Type":"ContainerStarted","Data":"d74bc94df3b3dcb54cebb3afc1eabb1a7c79356200d23972fc966edbf03ffb7f"}
Oct 31 04:06:55.303637 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:55.303604    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"
Oct 31 04:06:55.305164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:55.305137    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm" event={"ID":"d7190e98-41c5-426a-b7a3-ded4161c3266","Type":"ContainerStarted","Data":"084d2104fd5864660f49ae65a7826e0a582862aa329c68ed3940e0a51640c291"}
Oct 31 04:06:55.305270 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:55.305186    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm" event={"ID":"d7190e98-41c5-426a-b7a3-ded4161c3266","Type":"ContainerStarted","Data":"b227fd8410a6f97bd863042f557dce24d7ace986b0f7b75f7899049757b5c8cf"}
Oct 31 04:06:55.305270 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:55.305215    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"
Oct 31 04:06:55.417758 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:55.417650    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv" podStartSLOduration=1.4176361100000001 podStartE2EDuration="1.41763611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:55.415689378 +0000 UTC m=+3568.382022722" watchObservedRunningTime="2026-10-31 04:06:55.41763611 +0000 UTC m=+3568.383969455"
Oct 31 04:06:55.534089 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:55.534020    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm" podStartSLOduration=1.53399772 podStartE2EDuration="1.53399772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:55.528570258 +0000 UTC m=+3568.494903602" watchObservedRunningTime="2026-10-31 04:06:55.53399772 +0000 UTC m=+3568.500331077"
Oct 31 04:06:59.528001 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:06:59.527962    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:02.421714 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:02.421678    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:04.302454 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:04.302421    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:06.313907 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:06.313866    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"
Oct 31 04:07:06.314418 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:06.313947    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"
Oct 31 04:07:06.984826 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:07:06.984767    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:07:09.195288 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:09.195254    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:14.098191 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:14.098154    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:19.102017 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:19.101984    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:23.995463 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:23.995428    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:27.047103 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:27.047058    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"]
Oct 31 04:07:27.050438 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:27.050405    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-314fc8fe-thbgn"]
Oct 31 04:07:27.156638 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:27.156601    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b0a24fa-1f4d-440c-9d62-77182debb7e6" path="/var/lib/kubelet/pods/2b0a24fa-1f4d-440c-9d62-77182debb7e6/volumes"
Oct 31 04:07:28.618749 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:28.618711    4900 scope.go:122] "RemoveContainer" containerID="e6e581793d5f16dbccd29faa1103ac23b1d326e7e00a3def20754a24f805ae09"
Oct 31 04:07:28.628260 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:28.628235    4900 scope.go:122] "RemoveContainer" containerID="b6bb1e1a4666c5b7e95b98231c3bb674e0a7df821bd441f34f41eea6482b3851"
Oct 31 04:07:28.821979 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:28.821949    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:07:28.840384 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:28.840357    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:07:28.870572 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:28.870468    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:29.526219 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:29.526187    4900 generic.go:356] "Generic (PLEG): container finished" podID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerID="26113adcfa27137b1c1974bbea82b46e306b09b5e0cc6d6a556d1f38123e37f2" exitCode=0
Oct 31 04:07:29.526458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:29.526255    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p" event={"ID":"bc1a6b6b-c18f-40f6-b408-0090cafe29e1","Type":"ContainerDied","Data":"26113adcfa27137b1c1974bbea82b46e306b09b5e0cc6d6a556d1f38123e37f2"}
Oct 31 04:07:29.526719 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:29.526705    4900 scope.go:122] "RemoveContainer" containerID="26113adcfa27137b1c1974bbea82b46e306b09b5e0cc6d6a556d1f38123e37f2"
Oct 31 04:07:29.528455 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:29.528429    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:07:29.528525 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:29.528497    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:07:29.528561 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:29.528528    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:07:30.558870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:30.538903    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p" event={"ID":"bc1a6b6b-c18f-40f6-b408-0090cafe29e1","Type":"ContainerStarted","Data":"ff1bbfede1b3f17587662b373e8376832e92363b1479c6062245251f071b4084"}
Oct 31 04:07:31.186203 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:31.186167    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:07:31.186401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:31.186354    4900 patch_prober.go:29] interesting pod/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 04:07:31.186448 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:31.186429    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p" podUID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 04:07:31.509252 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:31.509214    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:32.186777 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:32.186741    4900 patch_prober.go:29] interesting pod/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 04:07:32.187221 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:32.186823    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p" podUID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 04:07:33.187235 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:33.187202    4900 patch_prober.go:29] interesting pod/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 04:07:33.187663 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:33.187253    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p" podUID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 04:07:33.774949 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:33.774915    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:34.186307 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:34.186228    4900 patch_prober.go:29] interesting pod/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 04:07:34.186307 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:34.186283    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p" podUID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 04:07:35.186298 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:35.186267    4900 patch_prober.go:29] interesting pod/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 31 04:07:35.186708 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:35.186324    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p" podUID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 31 04:07:36.190960 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:36.190925    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:07:36.191362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:36.191346    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:07:36.192061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:36.192037    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:07:36.986126 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:07:36.986088    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:07:37.089825 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:37.089776    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:37.826077 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:37.826043    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"]
Oct 31 04:07:37.937825 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:37.937778    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"]
Oct 31 04:07:38.270579 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:38.270542    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:38.470542 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:38.470503    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:38.934050 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:38.934017    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:42.551697 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:42.551633    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:42.827239 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:42.827145    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk" podUID="4e628d97-133e-4c25-b9b4-99346b0e6069" containerName="payload-processing" containerID="cri-o://f1434cc2d49474bdd5ee5260237d2cd3aea28c9b2efb2d285303c66adb5cee79" gracePeriod=25
Oct 31 04:07:42.938611 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:42.938566    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z" podUID="bbc6589f-a3bc-43dc-b642-a93a09016021" containerName="payload-pre-processing" containerID="cri-o://149218fa72bedc1399a6c7526c54320581fa221bc4d909b78c6cb0b371ce8d4d" gracePeriod=25
Oct 31 04:07:43.118083 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.118055    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"
Oct 31 04:07:43.225447 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.225422    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"
Oct 31 04:07:43.231021 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.230984    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e628d97-133e-4c25-b9b4-99346b0e6069-kube-api-access-ngczd\" (UniqueName: \"kubernetes.io/projected/4e628d97-133e-4c25-b9b4-99346b0e6069-kube-api-access-ngczd\") pod \"4e628d97-133e-4c25-b9b4-99346b0e6069\" (UID: \"4e628d97-133e-4c25-b9b4-99346b0e6069\") "
Oct 31 04:07:43.231176 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.231109    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e628d97-133e-4c25-b9b4-99346b0e6069-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e628d97-133e-4c25-b9b4-99346b0e6069-plugins-config-volume\") pod \"4e628d97-133e-4c25-b9b4-99346b0e6069\" (UID: \"4e628d97-133e-4c25-b9b4-99346b0e6069\") "
Oct 31 04:07:43.232437 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.232406    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e628d97-133e-4c25-b9b4-99346b0e6069-plugins-config-volume" pod "4e628d97-133e-4c25-b9b4-99346b0e6069" (UID: "4e628d97-133e-4c25-b9b4-99346b0e6069"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:07:43.233542 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.233521    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e628d97-133e-4c25-b9b4-99346b0e6069-kube-api-access-ngczd" pod "4e628d97-133e-4c25-b9b4-99346b0e6069" (UID: "4e628d97-133e-4c25-b9b4-99346b0e6069"). InnerVolumeSpecName "kube-api-access-ngczd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:07:43.333744 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.333648    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbc6589f-a3bc-43dc-b642-a93a09016021-kube-api-access-d7gm8\" (UniqueName: \"kubernetes.io/projected/bbc6589f-a3bc-43dc-b642-a93a09016021-kube-api-access-d7gm8\") pod \"bbc6589f-a3bc-43dc-b642-a93a09016021\" (UID: \"bbc6589f-a3bc-43dc-b642-a93a09016021\") "
Oct 31 04:07:43.333744 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.333742    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbc6589f-a3bc-43dc-b642-a93a09016021-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6589f-a3bc-43dc-b642-a93a09016021-plugins-config-volume\") pod \"bbc6589f-a3bc-43dc-b642-a93a09016021\" (UID: \"bbc6589f-a3bc-43dc-b642-a93a09016021\") "
Oct 31 04:07:43.334210 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.334167    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc6589f-a3bc-43dc-b642-a93a09016021-plugins-config-volume" pod "bbc6589f-a3bc-43dc-b642-a93a09016021" (UID: "bbc6589f-a3bc-43dc-b642-a93a09016021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:07:43.334854 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.334526    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e628d97-133e-4c25-b9b4-99346b0e6069-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:43.334854 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.334549    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngczd\" (UniqueName: \"kubernetes.io/projected/4e628d97-133e-4c25-b9b4-99346b0e6069-kube-api-access-ngczd\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:43.334854 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.334565    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6589f-a3bc-43dc-b642-a93a09016021-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:43.336054 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.336028    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc6589f-a3bc-43dc-b642-a93a09016021-kube-api-access-d7gm8" pod "bbc6589f-a3bc-43dc-b642-a93a09016021" (UID: "bbc6589f-a3bc-43dc-b642-a93a09016021"). InnerVolumeSpecName "kube-api-access-d7gm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:07:43.435829 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.435775    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7gm8\" (UniqueName: \"kubernetes.io/projected/bbc6589f-a3bc-43dc-b642-a93a09016021-kube-api-access-d7gm8\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:43.643803 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.643700    4900 generic.go:356] "Generic (PLEG): container finished" podID="4e628d97-133e-4c25-b9b4-99346b0e6069" containerID="f1434cc2d49474bdd5ee5260237d2cd3aea28c9b2efb2d285303c66adb5cee79" exitCode=0
Oct 31 04:07:43.643803 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.643764    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"
Oct 31 04:07:43.644383 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.643780    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk" event={"ID":"4e628d97-133e-4c25-b9b4-99346b0e6069","Type":"ContainerDied","Data":"f1434cc2d49474bdd5ee5260237d2cd3aea28c9b2efb2d285303c66adb5cee79"}
Oct 31 04:07:43.644383 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.643911    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk" event={"ID":"4e628d97-133e-4c25-b9b4-99346b0e6069","Type":"ContainerDied","Data":"51e50409715cbafc99b945142db687a49d12755d5d46ff1d35316bc2f0767699"}
Oct 31 04:07:43.644383 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.643947    4900 scope.go:122] "RemoveContainer" containerID="f1434cc2d49474bdd5ee5260237d2cd3aea28c9b2efb2d285303c66adb5cee79"
Oct 31 04:07:43.645218 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.645193    4900 generic.go:356] "Generic (PLEG): container finished" podID="bbc6589f-a3bc-43dc-b642-a93a09016021" containerID="149218fa72bedc1399a6c7526c54320581fa221bc4d909b78c6cb0b371ce8d4d" exitCode=0
Oct 31 04:07:43.645332 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.645243    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"
Oct 31 04:07:43.645332 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.645262    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z" event={"ID":"bbc6589f-a3bc-43dc-b642-a93a09016021","Type":"ContainerDied","Data":"149218fa72bedc1399a6c7526c54320581fa221bc4d909b78c6cb0b371ce8d4d"}
Oct 31 04:07:43.645332 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.645302    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z" event={"ID":"bbc6589f-a3bc-43dc-b642-a93a09016021","Type":"ContainerDied","Data":"5a0e9bcd0da1d3786e7f11e41f0a7805cd898766aa19de154c04effe3586ddd9"}
Oct 31 04:07:43.655095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.655048    4900 scope.go:122] "RemoveContainer" containerID="f1434cc2d49474bdd5ee5260237d2cd3aea28c9b2efb2d285303c66adb5cee79"
Oct 31 04:07:43.655375 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:07:43.655354    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1434cc2d49474bdd5ee5260237d2cd3aea28c9b2efb2d285303c66adb5cee79\": container with ID starting with f1434cc2d49474bdd5ee5260237d2cd3aea28c9b2efb2d285303c66adb5cee79 not found: ID does not exist" containerID="f1434cc2d49474bdd5ee5260237d2cd3aea28c9b2efb2d285303c66adb5cee79"
Oct 31 04:07:43.655469 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.655423    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1434cc2d49474bdd5ee5260237d2cd3aea28c9b2efb2d285303c66adb5cee79"} err="failed to get container status \"f1434cc2d49474bdd5ee5260237d2cd3aea28c9b2efb2d285303c66adb5cee79\": rpc error: code = NotFound desc = could not find container \"f1434cc2d49474bdd5ee5260237d2cd3aea28c9b2efb2d285303c66adb5cee79\": container with ID starting with f1434cc2d49474bdd5ee5260237d2cd3aea28c9b2efb2d285303c66adb5cee79 not found: ID does not exist"
Oct 31 04:07:43.655469 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.655455    4900 scope.go:122] "RemoveContainer" containerID="149218fa72bedc1399a6c7526c54320581fa221bc4d909b78c6cb0b371ce8d4d"
Oct 31 04:07:43.664591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.664569    4900 scope.go:122] "RemoveContainer" containerID="149218fa72bedc1399a6c7526c54320581fa221bc4d909b78c6cb0b371ce8d4d"
Oct 31 04:07:43.664921 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:07:43.664901    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"149218fa72bedc1399a6c7526c54320581fa221bc4d909b78c6cb0b371ce8d4d\": container with ID starting with 149218fa72bedc1399a6c7526c54320581fa221bc4d909b78c6cb0b371ce8d4d not found: ID does not exist" containerID="149218fa72bedc1399a6c7526c54320581fa221bc4d909b78c6cb0b371ce8d4d"
Oct 31 04:07:43.664977 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.664929    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"149218fa72bedc1399a6c7526c54320581fa221bc4d909b78c6cb0b371ce8d4d"} err="failed to get container status \"149218fa72bedc1399a6c7526c54320581fa221bc4d909b78c6cb0b371ce8d4d\": rpc error: code = NotFound desc = could not find container \"149218fa72bedc1399a6c7526c54320581fa221bc4d909b78c6cb0b371ce8d4d\": container with ID starting with 149218fa72bedc1399a6c7526c54320581fa221bc4d909b78c6cb0b371ce8d4d not found: ID does not exist"
Oct 31 04:07:43.808059 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.808029    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"]
Oct 31 04:07:43.811454 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.811431    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c7585c-bc54c9dc-zlb2z"]
Oct 31 04:07:43.920957 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.920882    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"]
Oct 31 04:07:43.924479 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:43.924456    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c7585c-c6bd6c7-xtwpk"]
Oct 31 04:07:44.079067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:44.079011    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:45.154220 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:45.154180    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e628d97-133e-4c25-b9b4-99346b0e6069" path="/var/lib/kubelet/pods/4e628d97-133e-4c25-b9b4-99346b0e6069/volumes"
Oct 31 04:07:45.154800 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:45.154771    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbc6589f-a3bc-43dc-b642-a93a09016021" path="/var/lib/kubelet/pods/bbc6589f-a3bc-43dc-b642-a93a09016021/volumes"
Oct 31 04:07:49.433770 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:49.433721    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:54.213858 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:54.213821    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:54.424159 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:54.424125    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:54.621154 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:54.621110    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:07:59.224194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:07:59.224153    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:02.750207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:02.750171    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:03.310210 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.310174    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"]
Oct 31 04:08:03.310406 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.310364    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc" podUID="3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3" containerName="maas-api" containerID="cri-o://3b3a007c5bec61e85fdd25a53911e67f85b7b2b8c9ca56292ef02f683f23617a" gracePeriod=30
Oct 31 04:08:03.484099 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.484067    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:03.586066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.586042    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:08:03.686838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.686813    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-openshift-service-ca\") pod \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") "
Oct 31 04:08:03.687018 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.687003    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-maas-api-tls\") pod \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") "
Oct 31 04:08:03.687067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.687052    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-metrics-certs\") pod \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") "
Oct 31 04:08:03.687117 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.687072    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-kube-api-access-blnp7\" (UniqueName: \"kubernetes.io/projected/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-kube-api-access-blnp7\") pod \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\" (UID: \"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3\") "
Oct 31 04:08:03.687179 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.687145    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-openshift-service-ca" pod "3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3" (UID: "3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:08:03.687598 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.687577    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:03.689103 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.689075    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-maas-api-tls" pod "3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3" (UID: "3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:08:03.689198 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.689106    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-metrics-certs" pod "3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3" (UID: "3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:08:03.689362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.689345    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-kube-api-access-blnp7" pod "3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3" (UID: "3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3"). InnerVolumeSpecName "kube-api-access-blnp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:08:03.772424 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.772397    4900 generic.go:356] "Generic (PLEG): container finished" podID="3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3" containerID="3b3a007c5bec61e85fdd25a53911e67f85b7b2b8c9ca56292ef02f683f23617a" exitCode=0
Oct 31 04:08:03.772865 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.772442    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"
Oct 31 04:08:03.772865 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.772487    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc" event={"ID":"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3","Type":"ContainerDied","Data":"3b3a007c5bec61e85fdd25a53911e67f85b7b2b8c9ca56292ef02f683f23617a"}
Oct 31 04:08:03.772865 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.772540    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc" event={"ID":"3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3","Type":"ContainerDied","Data":"0839756a2206b627673a78071df6928d47f3a46dd365c41bdc0fbce55d870abf"}
Oct 31 04:08:03.772865 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.772583    4900 scope.go:122] "RemoveContainer" containerID="3b3a007c5bec61e85fdd25a53911e67f85b7b2b8c9ca56292ef02f683f23617a"
Oct 31 04:08:03.785388 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.785365    4900 scope.go:122] "RemoveContainer" containerID="3b3a007c5bec61e85fdd25a53911e67f85b7b2b8c9ca56292ef02f683f23617a"
Oct 31 04:08:03.785709 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:08:03.785684    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b3a007c5bec61e85fdd25a53911e67f85b7b2b8c9ca56292ef02f683f23617a\": container with ID starting with 3b3a007c5bec61e85fdd25a53911e67f85b7b2b8c9ca56292ef02f683f23617a not found: ID does not exist" containerID="3b3a007c5bec61e85fdd25a53911e67f85b7b2b8c9ca56292ef02f683f23617a"
Oct 31 04:08:03.785780 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.785720    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3a007c5bec61e85fdd25a53911e67f85b7b2b8c9ca56292ef02f683f23617a"} err="failed to get container status \"3b3a007c5bec61e85fdd25a53911e67f85b7b2b8c9ca56292ef02f683f23617a\": rpc error: code = NotFound desc = could not find container \"3b3a007c5bec61e85fdd25a53911e67f85b7b2b8c9ca56292ef02f683f23617a\": container with ID starting with 3b3a007c5bec61e85fdd25a53911e67f85b7b2b8c9ca56292ef02f683f23617a not found: ID does not exist"
Oct 31 04:08:03.788901 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.788882    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:03.789002 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.788902    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:03.789002 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.788912    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blnp7\" (UniqueName: \"kubernetes.io/projected/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3-kube-api-access-blnp7\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:03.936592 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.936505    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"]
Oct 31 04:08:03.939673 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:03.939644    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc"]
Oct 31 04:08:05.153030 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:05.153000    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3" path="/var/lib/kubelet/pods/3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3/volumes"
Oct 31 04:08:05.303942 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:05.303909    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:06.575408 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:06.575372    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d8190b-5fb574b4fd-8brfc" podUID="3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.147:8443/health\": context deadline exceeded"
Oct 31 04:08:06.984892 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:08:06.984861    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:08:09.631726 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:09.631687    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:10.150585 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:10.150542    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:10.349010 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:10.348978    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:13.088329 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:13.088292    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:13.307764 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:13.307731    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:14.062479 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:14.062403    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:14.261508 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:14.261474    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:15.584076 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:15.584031    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:18.355855 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:18.355817    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:19.419421 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.419391    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"]
Oct 31 04:08:19.420050 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.420035    4900 cpu_manager.go:414] "containerMap: removing container" podUID="4e628d97-133e-4c25-b9b4-99346b0e6069" containerName="payload-processing"
Oct 31 04:08:19.420101 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.420050    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e628d97-133e-4c25-b9b4-99346b0e6069" containerName="payload-processing"
Oct 31 04:08:19.420101 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.420074    4900 cpu_manager.go:414] "containerMap: removing container" podUID="bbc6589f-a3bc-43dc-b642-a93a09016021" containerName="payload-pre-processing"
Oct 31 04:08:19.420101 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.420079    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbc6589f-a3bc-43dc-b642-a93a09016021" containerName="payload-pre-processing"
Oct 31 04:08:19.420202 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.420115    4900 cpu_manager.go:414] "containerMap: removing container" podUID="3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3" containerName="maas-api"
Oct 31 04:08:19.420202 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.420120    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3" containerName="maas-api"
Oct 31 04:08:19.420265 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.420211    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5aebad-ba5b-41e0-9df3-b6b4bdae08b3" containerName="maas-api"
Oct 31 04:08:19.420265 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.420226    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e628d97-133e-4c25-b9b4-99346b0e6069" containerName="payload-processing"
Oct 31 04:08:19.420265 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.420233    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbc6589f-a3bc-43dc-b642-a93a09016021" containerName="payload-pre-processing"
Oct 31 04:08:19.425022 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.425007    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:19.475743 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.475713    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"]
Oct 31 04:08:19.522964 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.522933    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 04:08:19.522964 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.522933    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-4ee9de5c5ba0\""
Oct 31 04:08:19.523178 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.522967    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 04:08:19.523178 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.522938    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 04:08:19.523178 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.522981    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-ddh5x\""
Oct 31 04:08:19.525677 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.525649    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-provider-credentials\") pod \"payload-processing-external-model-57fdb56688-j2l9b\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") " pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:19.525819 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.525726    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2npc\" (UniqueName: \"kubernetes.io/projected/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-kube-api-access-w2npc\") pod \"payload-processing-external-model-57fdb56688-j2l9b\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") " pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:19.525891 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.525874    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-plugins-config-volume\") pod \"payload-processing-external-model-57fdb56688-j2l9b\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") " pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:19.526030 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.526011    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-routing-overlay-volume\") pod \"payload-processing-external-model-57fdb56688-j2l9b\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") " pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:19.527374 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.527354    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 04:08:19.628161 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.628124    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-routing-overlay-volume\") pod \"payload-processing-external-model-57fdb56688-j2l9b\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") " pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:19.628333 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.628200    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-provider-credentials\") pod \"payload-processing-external-model-57fdb56688-j2l9b\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") " pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:19.628391 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.628331    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2npc\" (UniqueName: \"kubernetes.io/projected/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-kube-api-access-w2npc\") pod \"payload-processing-external-model-57fdb56688-j2l9b\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") " pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:19.628439 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.628426    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-plugins-config-volume\") pod \"payload-processing-external-model-57fdb56688-j2l9b\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") " pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:19.628924 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.628899    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-routing-overlay-volume\") pod \"payload-processing-external-model-57fdb56688-j2l9b\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") " pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:19.629331 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.629298    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-plugins-config-volume\") pod \"payload-processing-external-model-57fdb56688-j2l9b\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") " pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:19.630912 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.630884    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-provider-credentials\") pod \"payload-processing-external-model-57fdb56688-j2l9b\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") " pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:19.686120 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.686042    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2npc\" (UniqueName: \"kubernetes.io/projected/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-kube-api-access-w2npc\") pod \"payload-processing-external-model-57fdb56688-j2l9b\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") " pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:19.739195 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:19.739165    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:20.118880 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:20.118848    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"]
Oct 31 04:08:20.119929 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:08:20.119890    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c63f147_6918_48bb_9a6c_ceb080c3ad0c.slice/crio-ee18970741c28a9932a28b9ba7f9dd0f5508007869d6f5bc9c12192a5c7d5335 WatchSource:0}: Error finding container ee18970741c28a9932a28b9ba7f9dd0f5508007869d6f5bc9c12192a5c7d5335: Status 404 returned error can't find the container with id ee18970741c28a9932a28b9ba7f9dd0f5508007869d6f5bc9c12192a5c7d5335
Oct 31 04:08:20.428201 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:20.428115    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:20.736063 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:20.736032    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:20.932662 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:20.932616    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b" event={"ID":"0c63f147-6918-48bb-9a6c-ceb080c3ad0c","Type":"ContainerStarted","Data":"b947f184d22a87430f336e32d08132dc71a825b5a42409f9c1cb00794ffb0e0d"}
Oct 31 04:08:20.932878 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:20.932693    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b" event={"ID":"0c63f147-6918-48bb-9a6c-ceb080c3ad0c","Type":"ContainerStarted","Data":"ee18970741c28a9932a28b9ba7f9dd0f5508007869d6f5bc9c12192a5c7d5335"}
Oct 31 04:08:20.932878 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:20.932844    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:20.991275 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:20.991195    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:21.051709 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:21.051653    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b" podStartSLOduration=2.05163981 podStartE2EDuration="2.05163981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:08:21.049198174 +0000 UTC m=+3654.015531519" watchObservedRunningTime="2026-10-31 04:08:21.05163981 +0000 UTC m=+3654.017973154"
Oct 31 04:08:21.210038 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:21.210001    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:22.946753 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:22.946720    4900 generic.go:356] "Generic (PLEG): container finished" podID="0187126e-af1f-44bf-9cef-13217cb07672" containerID="69f01b93e95d8198bc82087298eff196179c18dba7f92ebc9d4d6904de6b10aa" exitCode=0
Oct 31 04:08:22.947212 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:22.946829    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s" event={"ID":"0187126e-af1f-44bf-9cef-13217cb07672","Type":"ContainerDied","Data":"69f01b93e95d8198bc82087298eff196179c18dba7f92ebc9d4d6904de6b10aa"}
Oct 31 04:08:22.947274 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:22.947259    4900 scope.go:122] "RemoveContainer" containerID="69f01b93e95d8198bc82087298eff196179c18dba7f92ebc9d4d6904de6b10aa"
Oct 31 04:08:22.948845 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:22.948818    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:08:22.948940 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:22.948887    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:08:22.948940 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:22.948927    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:08:23.965428 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:23.965391    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s" event={"ID":"0187126e-af1f-44bf-9cef-13217cb07672","Type":"ContainerStarted","Data":"31a2ee59594170621bdd9188a570c6858af3d9e91d04373ab067db7238ed391c"}
Oct 31 04:08:24.354816 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:24.350777    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:08:24.354816 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:24.350951    4900 patch_prober.go:29] interesting pod/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 31 04:08:24.354816 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:24.350996    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s" podUID="0187126e-af1f-44bf-9cef-13217cb07672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 31 04:08:25.100658 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:25.100616    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:25.341899 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:25.341863    4900 patch_prober.go:29] interesting pod/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 31 04:08:25.342178 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:25.342145    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s" podUID="0187126e-af1f-44bf-9cef-13217cb07672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 31 04:08:25.564241 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:25.564203    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:26.353814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:26.349177    4900 patch_prober.go:29] interesting pod/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 31 04:08:26.353814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:26.349240    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s" podUID="0187126e-af1f-44bf-9cef-13217cb07672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 31 04:08:27.359749 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:27.352121    4900 patch_prober.go:29] interesting pod/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 31 04:08:27.359749 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:27.352191    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s" podUID="0187126e-af1f-44bf-9cef-13217cb07672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 31 04:08:28.356817 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:28.356307    4900 patch_prober.go:29] interesting pod/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body=
Oct 31 04:08:28.356817 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:28.356379    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s" podUID="0187126e-af1f-44bf-9cef-13217cb07672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused"
Oct 31 04:08:28.668573 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:28.668497    4900 scope.go:122] "RemoveContainer" containerID="9792c7739a89c6672ad0cc3e2719de889e6a7335981efdd5b34f752c81f529fd"
Oct 31 04:08:29.345165 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:29.345133    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:08:29.345557 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:29.345540    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:08:29.346090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:29.346070    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:08:30.474850 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:30.474810    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:30.679284 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:30.679248    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:30.890457 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:30.890425    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:31.096862 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:31.096830    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:31.943383 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:31.943347    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:08:35.751033 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:35.750999    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:36.692268 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:36.692233    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:36.984419 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:08:36.984375    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:08:41.066160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:41.066124    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:46.161461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:46.161428    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"]
Oct 31 04:08:46.161933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:46.161634    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln" podUID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerName="istio-proxy" containerID="cri-o://a4cbc70c27cb81be2a475175979fe5dac713c9c8a36439a5736113284d975763" gracePeriod=30
Oct 31 04:08:46.210722 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:46.210691    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"]
Oct 31 04:08:46.210939 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:46.210919    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh" podUID="a1c901c9-0bd8-48ad-8642-04e39ba8e66d" containerName="istio-proxy" containerID="cri-o://f6002fb69cdec7cacbe52e261c4ec49419128231a3322f0f3c1b5f01358a29e2" gracePeriod=30
Oct 31 04:08:46.310480 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:46.310440    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:50.945202 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:50.945172    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:51.295254 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.295226    4900 patch_prober.go:29] interesting pod/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 04:08:51.295375 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.295285    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh" podUID="a1c901c9-0bd8-48ad-8642-04e39ba8e66d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 04:08:51.503860 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.503773    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:08:51.506603 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.506581    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:08:51.663916 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.663885    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/748dd79d-83b4-47cb-8282-847f3f4fa3de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748dd79d-83b4-47cb-8282-847f3f4fa3de-istiod-ca-cert\") pod \"748dd79d-83b4-47cb-8282-847f3f4fa3de\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") "
Oct 31 04:08:51.664064 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.663942    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-kube-api-access-h77gp\" (UniqueName: \"kubernetes.io/projected/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-kube-api-access-h77gp\") pod \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") "
Oct 31 04:08:51.664064 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.663976    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748dd79d-83b4-47cb-8282-847f3f4fa3de-kube-api-access-br6f2\" (UniqueName: \"kubernetes.io/projected/748dd79d-83b4-47cb-8282-847f3f4fa3de-kube-api-access-br6f2\") pod \"748dd79d-83b4-47cb-8282-847f3f4fa3de\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") "
Oct 31 04:08:51.664064 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664010    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-token\" (UniqueName: \"kubernetes.io/projected/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-token\") pod \"748dd79d-83b4-47cb-8282-847f3f4fa3de\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") "
Oct 31 04:08:51.664223 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664089    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-credential-socket\") pod \"748dd79d-83b4-47cb-8282-847f3f4fa3de\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") "
Oct 31 04:08:51.664223 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664121    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-podinfo\") pod \"748dd79d-83b4-47cb-8282-847f3f4fa3de\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") "
Oct 31 04:08:51.664223 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664149    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istiod-ca-cert\") pod \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") "
Oct 31 04:08:51.664223 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664177    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-workload-socket\") pod \"748dd79d-83b4-47cb-8282-847f3f4fa3de\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") "
Oct 31 04:08:51.664445 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664223    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-workload-certs\") pod \"748dd79d-83b4-47cb-8282-847f3f4fa3de\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") "
Oct 31 04:08:51.664445 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664266    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-data\") pod \"748dd79d-83b4-47cb-8282-847f3f4fa3de\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") "
Oct 31 04:08:51.664445 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664314    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-workload-socket\") pod \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") "
Oct 31 04:08:51.664445 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664344    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-workload-certs\") pod \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") "
Oct 31 04:08:51.664445 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664345    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748dd79d-83b4-47cb-8282-847f3f4fa3de-istiod-ca-cert" pod "748dd79d-83b4-47cb-8282-847f3f4fa3de" (UID: "748dd79d-83b4-47cb-8282-847f3f4fa3de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:08:51.664445 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664383    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-envoy\") pod \"748dd79d-83b4-47cb-8282-847f3f4fa3de\" (UID: \"748dd79d-83b4-47cb-8282-847f3f4fa3de\") "
Oct 31 04:08:51.664445 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664415    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-credential-socket" pod "748dd79d-83b4-47cb-8282-847f3f4fa3de" (UID: "748dd79d-83b4-47cb-8282-847f3f4fa3de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:51.664445 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664424    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-data\") pod \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") "
Oct 31 04:08:51.664929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664470    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-token\" (UniqueName: \"kubernetes.io/projected/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-token\") pod \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") "
Oct 31 04:08:51.664929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664502    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-envoy\") pod \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") "
Oct 31 04:08:51.664929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664541    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-credential-socket\") pod \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") "
Oct 31 04:08:51.664929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664568    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-podinfo\") pod \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\" (UID: \"a1c901c9-0bd8-48ad-8642-04e39ba8e66d\") "
Oct 31 04:08:51.664929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664594    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-workload-certs" pod "748dd79d-83b4-47cb-8282-847f3f4fa3de" (UID: "748dd79d-83b4-47cb-8282-847f3f4fa3de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:51.664929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664605    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-workload-socket" pod "a1c901c9-0bd8-48ad-8642-04e39ba8e66d" (UID: "a1c901c9-0bd8-48ad-8642-04e39ba8e66d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:51.664929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664532    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istiod-ca-cert" pod "a1c901c9-0bd8-48ad-8642-04e39ba8e66d" (UID: "a1c901c9-0bd8-48ad-8642-04e39ba8e66d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:08:51.664929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664813    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-data" pod "a1c901c9-0bd8-48ad-8642-04e39ba8e66d" (UID: "a1c901c9-0bd8-48ad-8642-04e39ba8e66d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:51.664929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.664829    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-workload-socket" pod "748dd79d-83b4-47cb-8282-847f3f4fa3de" (UID: "748dd79d-83b4-47cb-8282-847f3f4fa3de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:51.665419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.665029    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-workload-certs" pod "a1c901c9-0bd8-48ad-8642-04e39ba8e66d" (UID: "a1c901c9-0bd8-48ad-8642-04e39ba8e66d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:51.665419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.665260    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.665419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.665281    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.665419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.665296    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.665419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.665310    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.665419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.665323    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.665419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.665338    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.665419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.665355    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.665419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.665369    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/748dd79d-83b4-47cb-8282-847f3f4fa3de-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.665936 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.665423    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-data" pod "748dd79d-83b4-47cb-8282-847f3f4fa3de" (UID: "748dd79d-83b4-47cb-8282-847f3f4fa3de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:51.665936 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.665479    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-credential-socket" pod "a1c901c9-0bd8-48ad-8642-04e39ba8e66d" (UID: "a1c901c9-0bd8-48ad-8642-04e39ba8e66d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:51.666849 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.666820    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748dd79d-83b4-47cb-8282-847f3f4fa3de-kube-api-access-br6f2" pod "748dd79d-83b4-47cb-8282-847f3f4fa3de" (UID: "748dd79d-83b4-47cb-8282-847f3f4fa3de"). InnerVolumeSpecName "kube-api-access-br6f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:08:51.666960 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.666863    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-token" pod "748dd79d-83b4-47cb-8282-847f3f4fa3de" (UID: "748dd79d-83b4-47cb-8282-847f3f4fa3de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:08:51.667647 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.667609    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-podinfo" pod "a1c901c9-0bd8-48ad-8642-04e39ba8e66d" (UID: "a1c901c9-0bd8-48ad-8642-04e39ba8e66d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:08:51.667752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.667691    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-envoy" pod "748dd79d-83b4-47cb-8282-847f3f4fa3de" (UID: "748dd79d-83b4-47cb-8282-847f3f4fa3de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:51.667865 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.667751    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-podinfo" pod "748dd79d-83b4-47cb-8282-847f3f4fa3de" (UID: "748dd79d-83b4-47cb-8282-847f3f4fa3de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:08:51.667865 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.667819    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-token" pod "a1c901c9-0bd8-48ad-8642-04e39ba8e66d" (UID: "a1c901c9-0bd8-48ad-8642-04e39ba8e66d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:08:51.668093 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.668074    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-envoy" pod "a1c901c9-0bd8-48ad-8642-04e39ba8e66d" (UID: "a1c901c9-0bd8-48ad-8642-04e39ba8e66d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:08:51.668233 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.668212    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-kube-api-access-h77gp" pod "a1c901c9-0bd8-48ad-8642-04e39ba8e66d" (UID: "a1c901c9-0bd8-48ad-8642-04e39ba8e66d"). InnerVolumeSpecName "kube-api-access-h77gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:08:51.766306 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.766236    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.766306 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.766261    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.766306 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.766270    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.766306 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.766279    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/748dd79d-83b4-47cb-8282-847f3f4fa3de-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.766306 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.766290    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.766306 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.766305    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.766610 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.766314    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.766610 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.766323    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.766610 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.766331    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h77gp\" (UniqueName: \"kubernetes.io/projected/a1c901c9-0bd8-48ad-8642-04e39ba8e66d-kube-api-access-h77gp\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:51.766610 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:51.766341    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br6f2\" (UniqueName: \"kubernetes.io/projected/748dd79d-83b4-47cb-8282-847f3f4fa3de-kube-api-access-br6f2\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:08:52.213301 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.213214    4900 generic.go:356] "Generic (PLEG): container finished" podID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerID="a4cbc70c27cb81be2a475175979fe5dac713c9c8a36439a5736113284d975763" exitCode=0
Oct 31 04:08:52.213301 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.213260    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"
Oct 31 04:08:52.213814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.213254    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln" event={"ID":"748dd79d-83b4-47cb-8282-847f3f4fa3de","Type":"ContainerDied","Data":"a4cbc70c27cb81be2a475175979fe5dac713c9c8a36439a5736113284d975763"}
Oct 31 04:08:52.213814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.213379    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln" event={"ID":"748dd79d-83b4-47cb-8282-847f3f4fa3de","Type":"ContainerDied","Data":"24a6a3964dca8c66464886105537dd15ae91f567f85e42a953e18c6af94297bc"}
Oct 31 04:08:52.213814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.213405    4900 scope.go:122] "RemoveContainer" containerID="a4cbc70c27cb81be2a475175979fe5dac713c9c8a36439a5736113284d975763"
Oct 31 04:08:52.214738 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.214715    4900 generic.go:356] "Generic (PLEG): container finished" podID="a1c901c9-0bd8-48ad-8642-04e39ba8e66d" containerID="f6002fb69cdec7cacbe52e261c4ec49419128231a3322f0f3c1b5f01358a29e2" exitCode=0
Oct 31 04:08:52.214889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.214773    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"
Oct 31 04:08:52.214889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.214822    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh" event={"ID":"a1c901c9-0bd8-48ad-8642-04e39ba8e66d","Type":"ContainerDied","Data":"f6002fb69cdec7cacbe52e261c4ec49419128231a3322f0f3c1b5f01358a29e2"}
Oct 31 04:08:52.214889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.214867    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh" event={"ID":"a1c901c9-0bd8-48ad-8642-04e39ba8e66d","Type":"ContainerDied","Data":"94fdbc2bb9c4a7dfaef16082c3937242ab254b54c9f9b118af2682821e9dae22"}
Oct 31 04:08:52.224838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.224772    4900 scope.go:122] "RemoveContainer" containerID="d8edb4fa01bd797c1aff6d8568987c8a9f0911bc6f909c8e9788c03ba21ca22c"
Oct 31 04:08:52.248461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.248442    4900 scope.go:122] "RemoveContainer" containerID="a4cbc70c27cb81be2a475175979fe5dac713c9c8a36439a5736113284d975763"
Oct 31 04:08:52.248771 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:08:52.248747    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4cbc70c27cb81be2a475175979fe5dac713c9c8a36439a5736113284d975763\": container with ID starting with a4cbc70c27cb81be2a475175979fe5dac713c9c8a36439a5736113284d975763 not found: ID does not exist" containerID="a4cbc70c27cb81be2a475175979fe5dac713c9c8a36439a5736113284d975763"
Oct 31 04:08:52.248859 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.248778    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4cbc70c27cb81be2a475175979fe5dac713c9c8a36439a5736113284d975763"} err="failed to get container status \"a4cbc70c27cb81be2a475175979fe5dac713c9c8a36439a5736113284d975763\": rpc error: code = NotFound desc = could not find container \"a4cbc70c27cb81be2a475175979fe5dac713c9c8a36439a5736113284d975763\": container with ID starting with a4cbc70c27cb81be2a475175979fe5dac713c9c8a36439a5736113284d975763 not found: ID does not exist"
Oct 31 04:08:52.248859 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.248820    4900 scope.go:122] "RemoveContainer" containerID="d8edb4fa01bd797c1aff6d8568987c8a9f0911bc6f909c8e9788c03ba21ca22c"
Oct 31 04:08:52.249104 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:08:52.249086    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8edb4fa01bd797c1aff6d8568987c8a9f0911bc6f909c8e9788c03ba21ca22c\": container with ID starting with d8edb4fa01bd797c1aff6d8568987c8a9f0911bc6f909c8e9788c03ba21ca22c not found: ID does not exist" containerID="d8edb4fa01bd797c1aff6d8568987c8a9f0911bc6f909c8e9788c03ba21ca22c"
Oct 31 04:08:52.249144 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.249110    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8edb4fa01bd797c1aff6d8568987c8a9f0911bc6f909c8e9788c03ba21ca22c"} err="failed to get container status \"d8edb4fa01bd797c1aff6d8568987c8a9f0911bc6f909c8e9788c03ba21ca22c\": rpc error: code = NotFound desc = could not find container \"d8edb4fa01bd797c1aff6d8568987c8a9f0911bc6f909c8e9788c03ba21ca22c\": container with ID starting with d8edb4fa01bd797c1aff6d8568987c8a9f0911bc6f909c8e9788c03ba21ca22c not found: ID does not exist"
Oct 31 04:08:52.249144 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.249126    4900 scope.go:122] "RemoveContainer" containerID="f6002fb69cdec7cacbe52e261c4ec49419128231a3322f0f3c1b5f01358a29e2"
Oct 31 04:08:52.257681 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.257661    4900 scope.go:122] "RemoveContainer" containerID="f6002fb69cdec7cacbe52e261c4ec49419128231a3322f0f3c1b5f01358a29e2"
Oct 31 04:08:52.257938 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:08:52.257922    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6002fb69cdec7cacbe52e261c4ec49419128231a3322f0f3c1b5f01358a29e2\": container with ID starting with f6002fb69cdec7cacbe52e261c4ec49419128231a3322f0f3c1b5f01358a29e2 not found: ID does not exist" containerID="f6002fb69cdec7cacbe52e261c4ec49419128231a3322f0f3c1b5f01358a29e2"
Oct 31 04:08:52.258016 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.257942    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6002fb69cdec7cacbe52e261c4ec49419128231a3322f0f3c1b5f01358a29e2"} err="failed to get container status \"f6002fb69cdec7cacbe52e261c4ec49419128231a3322f0f3c1b5f01358a29e2\": rpc error: code = NotFound desc = could not find container \"f6002fb69cdec7cacbe52e261c4ec49419128231a3322f0f3c1b5f01358a29e2\": container with ID starting with f6002fb69cdec7cacbe52e261c4ec49419128231a3322f0f3c1b5f01358a29e2 not found: ID does not exist"
Oct 31 04:08:52.380255 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.380223    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"]
Oct 31 04:08:52.430316 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.430285    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-nl7ln"]
Oct 31 04:08:52.496809 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.496759    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"]
Oct 31 04:08:52.503728 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.503697    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c7585c-openshift-default-b5c7bbbf7-89czh"]
Oct 31 04:08:52.817612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:52.817534    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:53.153499 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:53.153421    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="748dd79d-83b4-47cb-8282-847f3f4fa3de" path="/var/lib/kubelet/pods/748dd79d-83b4-47cb-8282-847f3f4fa3de/volumes"
Oct 31 04:08:53.153924 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:53.153910    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c901c9-0bd8-48ad-8642-04e39ba8e66d" path="/var/lib/kubelet/pods/a1c901c9-0bd8-48ad-8642-04e39ba8e66d/volumes"
Oct 31 04:08:57.564796 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.564748    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"]
Oct 31 04:08:57.565343 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.565328    4900 cpu_manager.go:414] "containerMap: removing container" podUID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerName="istio-proxy"
Oct 31 04:08:57.565343 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.565343    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerName="istio-proxy"
Oct 31 04:08:57.565444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.565366    4900 cpu_manager.go:414] "containerMap: removing container" podUID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerName="istio-proxy"
Oct 31 04:08:57.565444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.565372    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerName="istio-proxy"
Oct 31 04:08:57.565444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.565381    4900 cpu_manager.go:414] "containerMap: removing container" podUID="a1c901c9-0bd8-48ad-8642-04e39ba8e66d" containerName="istio-proxy"
Oct 31 04:08:57.565444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.565386    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c901c9-0bd8-48ad-8642-04e39ba8e66d" containerName="istio-proxy"
Oct 31 04:08:57.565594 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.565462    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerName="istio-proxy"
Oct 31 04:08:57.565594 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.565474    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="748dd79d-83b4-47cb-8282-847f3f4fa3de" containerName="istio-proxy"
Oct 31 04:08:57.565594 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.565480    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c901c9-0bd8-48ad-8642-04e39ba8e66d" containerName="istio-proxy"
Oct 31 04:08:57.570335 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.570314    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"
Oct 31 04:08:57.628570 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.628544    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"]
Oct 31 04:08:57.675224 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.675194    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:08:57.675398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.675367    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:08:57.736560 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.736530    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jts95\" (UniqueName: \"kubernetes.io/projected/c410152f-0742-4ff7-87fa-71688aa7df32-kube-api-access-jts95\") pod \"maas-api-revoke-sub-c0b136c3-ds92v\" (UID: \"c410152f-0742-4ff7-87fa-71688aa7df32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"
Oct 31 04:08:57.736751 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.736568    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c410152f-0742-4ff7-87fa-71688aa7df32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0b136c3-ds92v\" (UID: \"c410152f-0742-4ff7-87fa-71688aa7df32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"
Oct 31 04:08:57.736868 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.736767    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c410152f-0742-4ff7-87fa-71688aa7df32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0b136c3-ds92v\" (UID: \"c410152f-0742-4ff7-87fa-71688aa7df32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"
Oct 31 04:08:57.792280 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.792249    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:57.837955 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.837880    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c410152f-0742-4ff7-87fa-71688aa7df32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0b136c3-ds92v\" (UID: \"c410152f-0742-4ff7-87fa-71688aa7df32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"
Oct 31 04:08:57.838099 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.838003    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jts95\" (UniqueName: \"kubernetes.io/projected/c410152f-0742-4ff7-87fa-71688aa7df32-kube-api-access-jts95\") pod \"maas-api-revoke-sub-c0b136c3-ds92v\" (UID: \"c410152f-0742-4ff7-87fa-71688aa7df32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"
Oct 31 04:08:57.838099 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.838028    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c410152f-0742-4ff7-87fa-71688aa7df32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0b136c3-ds92v\" (UID: \"c410152f-0742-4ff7-87fa-71688aa7df32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"
Oct 31 04:08:57.838497 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.838477    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c410152f-0742-4ff7-87fa-71688aa7df32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0b136c3-ds92v\" (UID: \"c410152f-0742-4ff7-87fa-71688aa7df32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"
Oct 31 04:08:57.840281 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.840261    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c410152f-0742-4ff7-87fa-71688aa7df32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0b136c3-ds92v\" (UID: \"c410152f-0742-4ff7-87fa-71688aa7df32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"
Oct 31 04:08:57.895662 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:57.895626    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jts95\" (UniqueName: \"kubernetes.io/projected/c410152f-0742-4ff7-87fa-71688aa7df32-kube-api-access-jts95\") pod \"maas-api-revoke-sub-c0b136c3-ds92v\" (UID: \"c410152f-0742-4ff7-87fa-71688aa7df32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"
Oct 31 04:08:58.018120 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:58.018090    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:08:58.183457 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:58.183375    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"
Oct 31 04:08:58.555840 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:08:58.555781    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc410152f_0742_4ff7_87fa_71688aa7df32.slice/crio-3590a89ca11cb956aa600f8a474dddfc807d074b0e04956596136d16372bdbfa WatchSource:0}: Error finding container 3590a89ca11cb956aa600f8a474dddfc807d074b0e04956596136d16372bdbfa: Status 404 returned error can't find the container with id 3590a89ca11cb956aa600f8a474dddfc807d074b0e04956596136d16372bdbfa
Oct 31 04:08:58.558200 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:58.558176    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"]
Oct 31 04:08:59.263537 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:59.263505    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v" event={"ID":"c410152f-0742-4ff7-87fa-71688aa7df32","Type":"ContainerStarted","Data":"71df5fb271c76804349383bc2c54c1edefa19834b9c85e47d555af66ea0802df"}
Oct 31 04:08:59.263966 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:59.263554    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v" event={"ID":"c410152f-0742-4ff7-87fa-71688aa7df32","Type":"ContainerStarted","Data":"3590a89ca11cb956aa600f8a474dddfc807d074b0e04956596136d16372bdbfa"}
Oct 31 04:08:59.378109 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:59.378054    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v" podStartSLOduration=2.378039439 podStartE2EDuration="2.378039439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:08:59.377531162 +0000 UTC m=+3692.343864518" watchObservedRunningTime="2026-10-31 04:08:59.378039439 +0000 UTC m=+3692.344372820"
Oct 31 04:08:59.660405 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:08:59.660314    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:00.270861 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:00.270819    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:00.271717 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:00.271687    4900 generic.go:356] "Generic (PLEG): container finished" podID="c410152f-0742-4ff7-87fa-71688aa7df32" containerID="71df5fb271c76804349383bc2c54c1edefa19834b9c85e47d555af66ea0802df" exitCode=0
Oct 31 04:09:00.271873 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:00.271769    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v" event={"ID":"c410152f-0742-4ff7-87fa-71688aa7df32","Type":"ContainerDied","Data":"71df5fb271c76804349383bc2c54c1edefa19834b9c85e47d555af66ea0802df"}
Oct 31 04:09:01.118859 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:01.118817    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:01.427419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:01.427393    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"
Oct 31 04:09:01.479392 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:01.479355    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c410152f-0742-4ff7-87fa-71688aa7df32-kube-api-access-jts95\" (UniqueName: \"kubernetes.io/projected/c410152f-0742-4ff7-87fa-71688aa7df32-kube-api-access-jts95\") pod \"c410152f-0742-4ff7-87fa-71688aa7df32\" (UID: \"c410152f-0742-4ff7-87fa-71688aa7df32\") "
Oct 31 04:09:01.479585 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:01.479404    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c410152f-0742-4ff7-87fa-71688aa7df32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c410152f-0742-4ff7-87fa-71688aa7df32-maas-api-openshift-ca\") pod \"c410152f-0742-4ff7-87fa-71688aa7df32\" (UID: \"c410152f-0742-4ff7-87fa-71688aa7df32\") "
Oct 31 04:09:01.479585 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:01.479513    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c410152f-0742-4ff7-87fa-71688aa7df32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c410152f-0742-4ff7-87fa-71688aa7df32-maas-api-xks-ca\") pod \"c410152f-0742-4ff7-87fa-71688aa7df32\" (UID: \"c410152f-0742-4ff7-87fa-71688aa7df32\") "
Oct 31 04:09:01.479921 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:01.479878    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c410152f-0742-4ff7-87fa-71688aa7df32-maas-api-openshift-ca" pod "c410152f-0742-4ff7-87fa-71688aa7df32" (UID: "c410152f-0742-4ff7-87fa-71688aa7df32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:01.480231 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:01.480212    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c410152f-0742-4ff7-87fa-71688aa7df32-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:01.481570 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:01.481544    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c410152f-0742-4ff7-87fa-71688aa7df32-kube-api-access-jts95" pod "c410152f-0742-4ff7-87fa-71688aa7df32" (UID: "c410152f-0742-4ff7-87fa-71688aa7df32"). InnerVolumeSpecName "kube-api-access-jts95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:01.481652 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:01.481575    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c410152f-0742-4ff7-87fa-71688aa7df32-maas-api-xks-ca" pod "c410152f-0742-4ff7-87fa-71688aa7df32" (UID: "c410152f-0742-4ff7-87fa-71688aa7df32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:09:01.581015 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:01.580981    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c410152f-0742-4ff7-87fa-71688aa7df32-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:01.581015 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:01.581011    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jts95\" (UniqueName: \"kubernetes.io/projected/c410152f-0742-4ff7-87fa-71688aa7df32-kube-api-access-jts95\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:02.285339 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:02.285306    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v" event={"ID":"c410152f-0742-4ff7-87fa-71688aa7df32","Type":"ContainerDied","Data":"3590a89ca11cb956aa600f8a474dddfc807d074b0e04956596136d16372bdbfa"}
Oct 31 04:09:02.285339 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:02.285331    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"
Oct 31 04:09:02.285549 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:02.285354    4900 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3590a89ca11cb956aa600f8a474dddfc807d074b0e04956596136d16372bdbfa"
Oct 31 04:09:03.115224 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:03.115175    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:03.400528 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:03.400436    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:06.883024 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:06.882985    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:06.984393 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:06.984358    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:09:07.675753 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:07.675718    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"]
Oct 31 04:09:08.008678 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:08.008594    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:08.217278 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:08.217248    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:08.678471 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:08.678432    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:09.894174 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:09.894098    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:10.084917 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:10.084880    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"]
Oct 31 04:09:10.101544 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:10.101517    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:10.306352 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:10.306323    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:12.556877 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:12.556830    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:12.676212 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:12.676174    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg" podUID="ab71a411-994b-41ac-a655-6c69e1be910d" containerName="payload-pre-processing" containerID="cri-o://0b28f4aa8ae2758eda9118b03e6cbcc2229a92d4ea648cdc98f82655f853899d" gracePeriod=25
Oct 31 04:09:12.764887 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:12.764853    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:12.969073 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:12.969045    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"
Oct 31 04:09:12.976212 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:12.976181    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:13.119571 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.119493    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab71a411-994b-41ac-a655-6c69e1be910d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab71a411-994b-41ac-a655-6c69e1be910d-plugins-config-volume\") pod \"ab71a411-994b-41ac-a655-6c69e1be910d\" (UID: \"ab71a411-994b-41ac-a655-6c69e1be910d\") "
Oct 31 04:09:13.119708 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.119692    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab71a411-994b-41ac-a655-6c69e1be910d-kube-api-access-6j8qh\" (UniqueName: \"kubernetes.io/projected/ab71a411-994b-41ac-a655-6c69e1be910d-kube-api-access-6j8qh\") pod \"ab71a411-994b-41ac-a655-6c69e1be910d\" (UID: \"ab71a411-994b-41ac-a655-6c69e1be910d\") "
Oct 31 04:09:13.119875 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.119852    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab71a411-994b-41ac-a655-6c69e1be910d-plugins-config-volume" pod "ab71a411-994b-41ac-a655-6c69e1be910d" (UID: "ab71a411-994b-41ac-a655-6c69e1be910d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:13.120079 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.120064    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab71a411-994b-41ac-a655-6c69e1be910d-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:13.121736 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.121716    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab71a411-994b-41ac-a655-6c69e1be910d-kube-api-access-6j8qh" pod "ab71a411-994b-41ac-a655-6c69e1be910d" (UID: "ab71a411-994b-41ac-a655-6c69e1be910d"). InnerVolumeSpecName "kube-api-access-6j8qh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:13.220941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.220906    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j8qh\" (UniqueName: \"kubernetes.io/projected/ab71a411-994b-41ac-a655-6c69e1be910d-kube-api-access-6j8qh\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:13.361377 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.361338    4900 generic.go:356] "Generic (PLEG): container finished" podID="ab71a411-994b-41ac-a655-6c69e1be910d" containerID="0b28f4aa8ae2758eda9118b03e6cbcc2229a92d4ea648cdc98f82655f853899d" exitCode=0
Oct 31 04:09:13.361592 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.361516    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg" event={"ID":"ab71a411-994b-41ac-a655-6c69e1be910d","Type":"ContainerDied","Data":"0b28f4aa8ae2758eda9118b03e6cbcc2229a92d4ea648cdc98f82655f853899d"}
Oct 31 04:09:13.361659 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.361600    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg" event={"ID":"ab71a411-994b-41ac-a655-6c69e1be910d","Type":"ContainerDied","Data":"5e30bc1333f762030d88617a78324fff93ef69c9c915683618fc0721736de6c6"}
Oct 31 04:09:13.361659 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.361631    4900 scope.go:122] "RemoveContainer" containerID="0b28f4aa8ae2758eda9118b03e6cbcc2229a92d4ea648cdc98f82655f853899d"
Oct 31 04:09:13.361828 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.361811    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"
Oct 31 04:09:13.377649 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.377624    4900 scope.go:122] "RemoveContainer" containerID="0b28f4aa8ae2758eda9118b03e6cbcc2229a92d4ea648cdc98f82655f853899d"
Oct 31 04:09:13.377988 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:13.377960    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b28f4aa8ae2758eda9118b03e6cbcc2229a92d4ea648cdc98f82655f853899d\": container with ID starting with 0b28f4aa8ae2758eda9118b03e6cbcc2229a92d4ea648cdc98f82655f853899d not found: ID does not exist" containerID="0b28f4aa8ae2758eda9118b03e6cbcc2229a92d4ea648cdc98f82655f853899d"
Oct 31 04:09:13.378110 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.377995    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b28f4aa8ae2758eda9118b03e6cbcc2229a92d4ea648cdc98f82655f853899d"} err="failed to get container status \"0b28f4aa8ae2758eda9118b03e6cbcc2229a92d4ea648cdc98f82655f853899d\": rpc error: code = NotFound desc = could not find container \"0b28f4aa8ae2758eda9118b03e6cbcc2229a92d4ea648cdc98f82655f853899d\": container with ID starting with 0b28f4aa8ae2758eda9118b03e6cbcc2229a92d4ea648cdc98f82655f853899d not found: ID does not exist"
Oct 31 04:09:13.526822 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.526776    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"]
Oct 31 04:09:13.530693 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:13.530669    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-1f1314-595c6dfbdc-qj6lg"]
Oct 31 04:09:14.389227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:14.389196    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:15.085329 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.085286    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b" podUID="0c63f147-6918-48bb-9a6c-ceb080c3ad0c" containerName="payload-processing" containerID="cri-o://b947f184d22a87430f336e32d08132dc71a825b5a42409f9c1cb00794ffb0e0d" gracePeriod=25
Oct 31 04:09:15.154539 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.154506    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab71a411-994b-41ac-a655-6c69e1be910d" path="/var/lib/kubelet/pods/ab71a411-994b-41ac-a655-6c69e1be910d/volumes"
Oct 31 04:09:15.355861 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.355801    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:09:15.376895 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.376864    4900 generic.go:356] "Generic (PLEG): container finished" podID="0c63f147-6918-48bb-9a6c-ceb080c3ad0c" containerID="b947f184d22a87430f336e32d08132dc71a825b5a42409f9c1cb00794ffb0e0d" exitCode=0
Oct 31 04:09:15.377020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.376904    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"
Oct 31 04:09:15.377020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.376950    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b" event={"ID":"0c63f147-6918-48bb-9a6c-ceb080c3ad0c","Type":"ContainerDied","Data":"b947f184d22a87430f336e32d08132dc71a825b5a42409f9c1cb00794ffb0e0d"}
Oct 31 04:09:15.377020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.377010    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b" event={"ID":"0c63f147-6918-48bb-9a6c-ceb080c3ad0c","Type":"ContainerDied","Data":"ee18970741c28a9932a28b9ba7f9dd0f5508007869d6f5bc9c12192a5c7d5335"}
Oct 31 04:09:15.377132 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.377024    4900 scope.go:122] "RemoveContainer" containerID="b947f184d22a87430f336e32d08132dc71a825b5a42409f9c1cb00794ffb0e0d"
Oct 31 04:09:15.385996 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.385978    4900 scope.go:122] "RemoveContainer" containerID="b947f184d22a87430f336e32d08132dc71a825b5a42409f9c1cb00794ffb0e0d"
Oct 31 04:09:15.386268 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:15.386245    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b947f184d22a87430f336e32d08132dc71a825b5a42409f9c1cb00794ffb0e0d\": container with ID starting with b947f184d22a87430f336e32d08132dc71a825b5a42409f9c1cb00794ffb0e0d not found: ID does not exist" containerID="b947f184d22a87430f336e32d08132dc71a825b5a42409f9c1cb00794ffb0e0d"
Oct 31 04:09:15.386364 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.386273    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b947f184d22a87430f336e32d08132dc71a825b5a42409f9c1cb00794ffb0e0d"} err="failed to get container status \"b947f184d22a87430f336e32d08132dc71a825b5a42409f9c1cb00794ffb0e0d\": rpc error: code = NotFound desc = could not find container \"b947f184d22a87430f336e32d08132dc71a825b5a42409f9c1cb00794ffb0e0d\": container with ID starting with b947f184d22a87430f336e32d08132dc71a825b5a42409f9c1cb00794ffb0e0d not found: ID does not exist"
Oct 31 04:09:15.445350 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.445323    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-kube-api-access-w2npc\" (UniqueName: \"kubernetes.io/projected/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-kube-api-access-w2npc\") pod \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") "
Oct 31 04:09:15.445722 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.445412    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-provider-credentials\") pod \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") "
Oct 31 04:09:15.445722 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.445474    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-plugins-config-volume\") pod \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") "
Oct 31 04:09:15.445722 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.445606    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-routing-overlay-volume\") pod \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\" (UID: \"0c63f147-6918-48bb-9a6c-ceb080c3ad0c\") "
Oct 31 04:09:15.445945 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.445920    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-routing-overlay-volume" pod "0c63f147-6918-48bb-9a6c-ceb080c3ad0c" (UID: "0c63f147-6918-48bb-9a6c-ceb080c3ad0c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:15.446015 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.445940    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-plugins-config-volume" pod "0c63f147-6918-48bb-9a6c-ceb080c3ad0c" (UID: "0c63f147-6918-48bb-9a6c-ceb080c3ad0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:15.446173 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.446155    4900 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-routing-overlay-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:15.446243 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.446173    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:15.447601 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.447577    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-kube-api-access-w2npc" pod "0c63f147-6918-48bb-9a6c-ceb080c3ad0c" (UID: "0c63f147-6918-48bb-9a6c-ceb080c3ad0c"). InnerVolumeSpecName "kube-api-access-w2npc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:15.447718 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.447655    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-provider-credentials" pod "0c63f147-6918-48bb-9a6c-ceb080c3ad0c" (UID: "0c63f147-6918-48bb-9a6c-ceb080c3ad0c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:15.547435 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.547396    4900 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-provider-credentials\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:15.547435 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.547426    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2npc\" (UniqueName: \"kubernetes.io/projected/0c63f147-6918-48bb-9a6c-ceb080c3ad0c-kube-api-access-w2npc\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:15.811956 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.811921    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"]
Oct 31 04:09:15.821797 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:15.821755    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-57fdb56688-j2l9b"]
Oct 31 04:09:16.455906 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:16.455871    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:17.155403 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.155368    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c63f147-6918-48bb-9a6c-ceb080c3ad0c" path="/var/lib/kubelet/pods/0c63f147-6918-48bb-9a6c-ceb080c3ad0c/volumes"
Oct 31 04:09:17.542488 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.542455    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"]
Oct 31 04:09:17.543080 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.543064    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c410152f-0742-4ff7-87fa-71688aa7df32" containerName="revoke-keys"
Oct 31 04:09:17.543123 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.543081    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c410152f-0742-4ff7-87fa-71688aa7df32" containerName="revoke-keys"
Oct 31 04:09:17.543123 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.543099    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0c63f147-6918-48bb-9a6c-ceb080c3ad0c" containerName="payload-processing"
Oct 31 04:09:17.543123 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.543104    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c63f147-6918-48bb-9a6c-ceb080c3ad0c" containerName="payload-processing"
Oct 31 04:09:17.543123 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.543115    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ab71a411-994b-41ac-a655-6c69e1be910d" containerName="payload-pre-processing"
Oct 31 04:09:17.543123 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.543121    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab71a411-994b-41ac-a655-6c69e1be910d" containerName="payload-pre-processing"
Oct 31 04:09:17.543284 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.543239    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c410152f-0742-4ff7-87fa-71688aa7df32" containerName="revoke-keys"
Oct 31 04:09:17.543284 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.543247    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c63f147-6918-48bb-9a6c-ceb080c3ad0c" containerName="payload-processing"
Oct 31 04:09:17.543284 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.543256    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab71a411-994b-41ac-a655-6c69e1be910d" containerName="payload-pre-processing"
Oct 31 04:09:17.547949 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.547934    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:17.601194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.601148    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"]
Oct 31 04:09:17.647078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.647042    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 04:09:17.647078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.647060    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 04:09:17.647078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.647055    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 04:09:17.647320 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.647057    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3070c2924b04\""
Oct 31 04:09:17.647320 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.647041    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 04:09:17.650264 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.650245    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-rpvnj\""
Oct 31 04:09:17.671052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.671027    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ad9902-7094-4f57-939e-521d3117e0b5-routing-overlay-volume\") pod \"payload-processing-external-model-6b9b48ccc9-chpnh\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:17.671178 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.671161    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzvtl\" (UniqueName: \"kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-kube-api-access-fzvtl\") pod \"payload-processing-external-model-6b9b48ccc9-chpnh\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:17.671231 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.671220    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials\") pod \"payload-processing-external-model-6b9b48ccc9-chpnh\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:17.671319 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.671303    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ad9902-7094-4f57-939e-521d3117e0b5-plugins-config-volume\") pod \"payload-processing-external-model-6b9b48ccc9-chpnh\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:17.772612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.772571    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ad9902-7094-4f57-939e-521d3117e0b5-routing-overlay-volume\") pod \"payload-processing-external-model-6b9b48ccc9-chpnh\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:17.772862 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.772702    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzvtl\" (UniqueName: \"kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-kube-api-access-fzvtl\") pod \"payload-processing-external-model-6b9b48ccc9-chpnh\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:17.772862 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.772756    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials\") pod \"payload-processing-external-model-6b9b48ccc9-chpnh\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:17.772862 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.772836    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ad9902-7094-4f57-939e-521d3117e0b5-plugins-config-volume\") pod \"payload-processing-external-model-6b9b48ccc9-chpnh\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:17.773413 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.773383    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ad9902-7094-4f57-939e-521d3117e0b5-routing-overlay-volume\") pod \"payload-processing-external-model-6b9b48ccc9-chpnh\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:17.773602 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.773584    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ad9902-7094-4f57-939e-521d3117e0b5-plugins-config-volume\") pod \"payload-processing-external-model-6b9b48ccc9-chpnh\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:17.775130 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.775108    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials\") pod \"payload-processing-external-model-6b9b48ccc9-chpnh\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:17.830214 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.830145    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzvtl\" (UniqueName: \"kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-kube-api-access-fzvtl\") pod \"payload-processing-external-model-6b9b48ccc9-chpnh\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:17.861401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:17.861374    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:18.229044 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:09:18.229010    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ad9902_7094_4f57_939e_521d3117e0b5.slice/crio-1eeba67770a34a4d57b0ce6d40ab50f15c145ce0992b1bf1a1c609661c13d799 WatchSource:0}: Error finding container 1eeba67770a34a4d57b0ce6d40ab50f15c145ce0992b1bf1a1c609661c13d799: Status 404 returned error can't find the container with id 1eeba67770a34a4d57b0ce6d40ab50f15c145ce0992b1bf1a1c609661c13d799
Oct 31 04:09:18.231629 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:18.231576    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"]
Oct 31 04:09:18.399528 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:18.399489    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh" event={"ID":"19ad9902-7094-4f57-939e-521d3117e0b5","Type":"ContainerStarted","Data":"fa03967b8a48b2d598d0c27f5c3ad15b026e79276b4718a6f0f6f65c02131dd7"}
Oct 31 04:09:18.399713 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:18.399553    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:18.399713 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:18.399571    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh" event={"ID":"19ad9902-7094-4f57-939e-521d3117e0b5","Type":"ContainerStarted","Data":"1eeba67770a34a4d57b0ce6d40ab50f15c145ce0992b1bf1a1c609661c13d799"}
Oct 31 04:09:18.515244 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:18.515197    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh" podStartSLOduration=1.515182866 podStartE2EDuration="1.515182866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:18.512850819 +0000 UTC m=+3711.479184162" watchObservedRunningTime="2026-10-31 04:09:18.515182866 +0000 UTC m=+3711.481516233"
Oct 31 04:09:18.866586 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:18.866500    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:19.067774 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:19.067736    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:19.369150 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:19.369114    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:19.499346 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:19.499316    4900 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3070c2924b04: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:19.499346 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:19.499342    4900 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:19.499566 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:19.499400    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials podName:19ad9902-7094-4f57-939e-521d3117e0b5 nodeName:}" failed. No retries permitted until 2026-10-31 04:09:19.999385625 +0000 UTC m=+3712.965718946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials") pod "payload-processing-external-model-6b9b48ccc9-chpnh" (UID: "19ad9902-7094-4f57-939e-521d3117e0b5") : secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:19.638826 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:19.638709    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:20.005302 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:20.005267    4900 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3070c2924b04: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:20.005302 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:20.005297    4900 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:20.005764 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:20.005351    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials podName:19ad9902-7094-4f57-939e-521d3117e0b5 nodeName:}" failed. No retries permitted until 2026-10-31 04:09:21.00533675 +0000 UTC m=+3713.971670103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials") pod "payload-processing-external-model-6b9b48ccc9-chpnh" (UID: "19ad9902-7094-4f57-939e-521d3117e0b5") : secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:20.343291 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:20.343209    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"]
Oct 31 04:09:20.565255 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:20.565217    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:21.018759 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:21.018726    4900 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3070c2924b04: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:21.018759 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:21.018752    4900 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:21.019230 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:21.018843    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials podName:19ad9902-7094-4f57-939e-521d3117e0b5 nodeName:}" failed. No retries permitted until 2026-10-31 04:09:23.018829098 +0000 UTC m=+3715.985162421 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials") pod "payload-processing-external-model-6b9b48ccc9-chpnh" (UID: "19ad9902-7094-4f57-939e-521d3117e0b5") : secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:23.048157 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:23.048122    4900 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3070c2924b04: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:23.048157 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:23.048152    4900 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:23.048678 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:23.048211    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials podName:19ad9902-7094-4f57-939e-521d3117e0b5 nodeName:}" failed. No retries permitted until 2026-10-31 04:09:27.048195697 +0000 UTC m=+3720.014529033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials") pod "payload-processing-external-model-6b9b48ccc9-chpnh" (UID: "19ad9902-7094-4f57-939e-521d3117e0b5") : secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:24.307079 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:24.307036    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:25.154814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.154705    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:25.408330 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.408241    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:25.411177 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.411150    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh" podUID="19ad9902-7094-4f57-939e-521d3117e0b5" containerName="payload-processing" containerID="cri-o://fa03967b8a48b2d598d0c27f5c3ad15b026e79276b4718a6f0f6f65c02131dd7" gracePeriod=25
Oct 31 04:09:25.700830 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.700799    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:25.785988 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.785957    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials\") pod \"19ad9902-7094-4f57-939e-521d3117e0b5\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") "
Oct 31 04:09:25.785988 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.785992    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ad9902-7094-4f57-939e-521d3117e0b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ad9902-7094-4f57-939e-521d3117e0b5-plugins-config-volume\") pod \"19ad9902-7094-4f57-939e-521d3117e0b5\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") "
Oct 31 04:09:25.786199 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.786034    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ad9902-7094-4f57-939e-521d3117e0b5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ad9902-7094-4f57-939e-521d3117e0b5-routing-overlay-volume\") pod \"19ad9902-7094-4f57-939e-521d3117e0b5\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") "
Oct 31 04:09:25.786199 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.786134    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-kube-api-access-fzvtl\" (UniqueName: \"kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-kube-api-access-fzvtl\") pod \"19ad9902-7094-4f57-939e-521d3117e0b5\" (UID: \"19ad9902-7094-4f57-939e-521d3117e0b5\") "
Oct 31 04:09:25.786466 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.786440    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ad9902-7094-4f57-939e-521d3117e0b5-plugins-config-volume" pod "19ad9902-7094-4f57-939e-521d3117e0b5" (UID: "19ad9902-7094-4f57-939e-521d3117e0b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:25.786553 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.786503    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ad9902-7094-4f57-939e-521d3117e0b5-routing-overlay-volume" pod "19ad9902-7094-4f57-939e-521d3117e0b5" (UID: "19ad9902-7094-4f57-939e-521d3117e0b5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:25.786680 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.786662    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ad9902-7094-4f57-939e-521d3117e0b5-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:25.786742 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.786685    4900 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ad9902-7094-4f57-939e-521d3117e0b5-routing-overlay-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:25.788243 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.788218    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials" pod "19ad9902-7094-4f57-939e-521d3117e0b5" (UID: "19ad9902-7094-4f57-939e-521d3117e0b5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:25.788367 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.788266    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-kube-api-access-fzvtl" pod "19ad9902-7094-4f57-939e-521d3117e0b5" (UID: "19ad9902-7094-4f57-939e-521d3117e0b5"). InnerVolumeSpecName "kube-api-access-fzvtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:25.826451 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.826414    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:25.887556 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.887521    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzvtl\" (UniqueName: \"kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-kube-api-access-fzvtl\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:25.887556 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:25.887551    4900 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ad9902-7094-4f57-939e-521d3117e0b5-provider-credentials\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:26.031538 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:26.031498    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:26.447656 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:26.447567    4900 generic.go:356] "Generic (PLEG): container finished" podID="19ad9902-7094-4f57-939e-521d3117e0b5" containerID="fa03967b8a48b2d598d0c27f5c3ad15b026e79276b4718a6f0f6f65c02131dd7" exitCode=0
Oct 31 04:09:26.447656 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:26.447620    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"
Oct 31 04:09:26.448153 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:26.447653    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh" event={"ID":"19ad9902-7094-4f57-939e-521d3117e0b5","Type":"ContainerDied","Data":"fa03967b8a48b2d598d0c27f5c3ad15b026e79276b4718a6f0f6f65c02131dd7"}
Oct 31 04:09:26.448153 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:26.447708    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh" event={"ID":"19ad9902-7094-4f57-939e-521d3117e0b5","Type":"ContainerDied","Data":"1eeba67770a34a4d57b0ce6d40ab50f15c145ce0992b1bf1a1c609661c13d799"}
Oct 31 04:09:26.448153 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:26.447728    4900 scope.go:122] "RemoveContainer" containerID="fa03967b8a48b2d598d0c27f5c3ad15b026e79276b4718a6f0f6f65c02131dd7"
Oct 31 04:09:26.458198 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:26.458180    4900 scope.go:122] "RemoveContainer" containerID="fa03967b8a48b2d598d0c27f5c3ad15b026e79276b4718a6f0f6f65c02131dd7"
Oct 31 04:09:26.458508 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:26.458489    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa03967b8a48b2d598d0c27f5c3ad15b026e79276b4718a6f0f6f65c02131dd7\": container with ID starting with fa03967b8a48b2d598d0c27f5c3ad15b026e79276b4718a6f0f6f65c02131dd7 not found: ID does not exist" containerID="fa03967b8a48b2d598d0c27f5c3ad15b026e79276b4718a6f0f6f65c02131dd7"
Oct 31 04:09:26.458552 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:26.458516    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa03967b8a48b2d598d0c27f5c3ad15b026e79276b4718a6f0f6f65c02131dd7"} err="failed to get container status \"fa03967b8a48b2d598d0c27f5c3ad15b026e79276b4718a6f0f6f65c02131dd7\": rpc error: code = NotFound desc = could not find container \"fa03967b8a48b2d598d0c27f5c3ad15b026e79276b4718a6f0f6f65c02131dd7\": container with ID starting with fa03967b8a48b2d598d0c27f5c3ad15b026e79276b4718a6f0f6f65c02131dd7 not found: ID does not exist"
Oct 31 04:09:26.612297 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:26.612264    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"]
Oct 31 04:09:26.662532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:26.662501    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9b48ccc9-chpnh"]
Oct 31 04:09:27.154734 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:27.154699    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ad9902-7094-4f57-939e-521d3117e0b5" path="/var/lib/kubelet/pods/19ad9902-7094-4f57-939e-521d3117e0b5/volumes"
Oct 31 04:09:28.094534 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.094497    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"]
Oct 31 04:09:28.095156 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.095138    4900 cpu_manager.go:414] "containerMap: removing container" podUID="19ad9902-7094-4f57-939e-521d3117e0b5" containerName="payload-processing"
Oct 31 04:09:28.095204 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.095157    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ad9902-7094-4f57-939e-521d3117e0b5" containerName="payload-processing"
Oct 31 04:09:28.095283 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.095273    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ad9902-7094-4f57-939e-521d3117e0b5" containerName="payload-processing"
Oct 31 04:09:28.099934 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.099903    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.202305 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.202257    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-96js4\""
Oct 31 04:09:28.205567 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.205536    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"]
Oct 31 04:09:28.219731 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.219691    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.219912 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.219848    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.219912 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.219888    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.220048 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.220017    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.220100 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.220084    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.220223 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.220203    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkbcl\" (UniqueName: \"kubernetes.io/projected/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-kube-api-access-gkbcl\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.220298 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.220238    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.220356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.220329    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.220420 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.220393    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.321992 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.321953    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbcl\" (UniqueName: \"kubernetes.io/projected/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-kube-api-access-gkbcl\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.322206 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.321994    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.322206 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.322077    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.322206 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.322137    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.322206 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.322197    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.322422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.322263    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.322422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.322291    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.322422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.322337    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.322422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.322381    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.322643 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.322595    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.322643 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.322620    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.322748 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.322680    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.322925 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.322903    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.323169 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.323149    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.325153 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.325127    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.325611 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.325582    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.379102 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.379025    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkbcl\" (UniqueName: \"kubernetes.io/projected/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-kube-api-access-gkbcl\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.379268 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.379227    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7f8ce75-4372-4c11-aea6-6922ed91ba8a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-lxq9w\" (UID: \"d7f8ce75-4372-4c11-aea6-6922ed91ba8a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.415461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.415425    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:28.853538 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:09:28.853508    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f8ce75_4372_4c11_aea6_6922ed91ba8a.slice/crio-6e8a6631d303016512df209ef9167f1b9c6a48d2c3d150473fba4e11945d3ad1 WatchSource:0}: Error finding container 6e8a6631d303016512df209ef9167f1b9c6a48d2c3d150473fba4e11945d3ad1: Status 404 returned error can't find the container with id 6e8a6631d303016512df209ef9167f1b9c6a48d2c3d150473fba4e11945d3ad1
Oct 31 04:09:28.856076 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.856051    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"]
Oct 31 04:09:28.856272 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.856243    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:09:28.856326 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.856312    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:09:28.856363 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:28.856346    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:09:29.193032 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:29.192990    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:29.476748 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:29.475462    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" event={"ID":"d7f8ce75-4372-4c11-aea6-6922ed91ba8a","Type":"ContainerStarted","Data":"3e94bf1d2dcb34bedaa13a2905f0fa0e8f1ff9419cdebecdc5736d0a32fbaa38"}
Oct 31 04:09:29.476748 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:29.475555    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" event={"ID":"d7f8ce75-4372-4c11-aea6-6922ed91ba8a","Type":"ContainerStarted","Data":"6e8a6631d303016512df209ef9167f1b9c6a48d2c3d150473fba4e11945d3ad1"}
Oct 31 04:09:29.593379 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:29.593330    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" podStartSLOduration=1.593314603 podStartE2EDuration="1.593314603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:29.592409556 +0000 UTC m=+3722.558742900" watchObservedRunningTime="2026-10-31 04:09:29.593314603 +0000 UTC m=+3722.559647947"
Oct 31 04:09:29.808910 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:29.808877    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:30.277322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.277284    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"]
Oct 31 04:09:30.281335 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.281317    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:30.354819 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.336129    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"]
Oct 31 04:09:30.359809 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.355193    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 04:09:30.359809 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.355343    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-dlznn\""
Oct 31 04:09:30.359809 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.355416    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3070c2924b04\""
Oct 31 04:09:30.359809 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.355472    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 04:09:30.359809 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.355548    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 04:09:30.359809 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.355605    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 04:09:30.385578 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.385543    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-routing-overlay-volume\") pod \"payload-processing-external-model-6b9b48ccc9-qt58k\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:30.385730 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.385613    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume\") pod \"payload-processing-external-model-6b9b48ccc9-qt58k\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:30.385730 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.385632    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hdmv\" (UniqueName: \"kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-kube-api-access-8hdmv\") pod \"payload-processing-external-model-6b9b48ccc9-qt58k\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:30.385831 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.385800    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials\") pod \"payload-processing-external-model-6b9b48ccc9-qt58k\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:30.416253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.416220    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:30.417996 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.417972    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lxq9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 31 04:09:30.418162 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.418144    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" podUID="d7f8ce75-4372-4c11-aea6-6922ed91ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 31 04:09:30.487182 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.487140    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-routing-overlay-volume\") pod \"payload-processing-external-model-6b9b48ccc9-qt58k\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:30.487392 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.487251    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume\") pod \"payload-processing-external-model-6b9b48ccc9-qt58k\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:30.487392 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.487283    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hdmv\" (UniqueName: \"kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-kube-api-access-8hdmv\") pod \"payload-processing-external-model-6b9b48ccc9-qt58k\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:30.487392 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.487342    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials\") pod \"payload-processing-external-model-6b9b48ccc9-qt58k\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:30.492914 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.492868    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-routing-overlay-volume\") pod \"payload-processing-external-model-6b9b48ccc9-qt58k\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:30.493192 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.493173    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume\") pod \"payload-processing-external-model-6b9b48ccc9-qt58k\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:30.493748 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.493726    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials\") pod \"payload-processing-external-model-6b9b48ccc9-qt58k\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:30.562562 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.562476    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hdmv\" (UniqueName: \"kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-kube-api-access-8hdmv\") pod \"payload-processing-external-model-6b9b48ccc9-qt58k\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") " pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:30.595137 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.595100    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:30.990978 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:09:30.990951    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd723be65_afaf_4734_ada6_f8bf362ddf10.slice/crio-a9a5feb61981297448a3d2b9168b0967337258d33c4555906f938ed2681d6971 WatchSource:0}: Error finding container a9a5feb61981297448a3d2b9168b0967337258d33c4555906f938ed2681d6971: Status 404 returned error can't find the container with id a9a5feb61981297448a3d2b9168b0967337258d33c4555906f938ed2681d6971
Oct 31 04:09:30.993548 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:30.993525    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"]
Oct 31 04:09:31.010591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:31.010571    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:31.416939 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:31.416850    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lxq9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 31 04:09:31.416939 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:31.416912    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" podUID="d7f8ce75-4372-4c11-aea6-6922ed91ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 31 04:09:31.492535 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:31.492494    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k" event={"ID":"d723be65-afaf-4734-ada6-f8bf362ddf10","Type":"ContainerStarted","Data":"0bc607213db5a3ad366b58c834786cb62cb4830d27b32bf4c050ae5f0cd3750a"}
Oct 31 04:09:31.492729 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:31.492558    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k" event={"ID":"d723be65-afaf-4734-ada6-f8bf362ddf10","Type":"ContainerStarted","Data":"a9a5feb61981297448a3d2b9168b0967337258d33c4555906f938ed2681d6971"}
Oct 31 04:09:31.602255 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:31.602158    4900 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3070c2924b04: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:31.602255 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:31.602180    4900 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:31.602255 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:31.602233    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials podName:d723be65-afaf-4734-ada6-f8bf362ddf10 nodeName:}" failed. No retries permitted until 2026-10-31 04:09:32.102213142 +0000 UTC m=+3725.068546467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials") pod "payload-processing-external-model-6b9b48ccc9-qt58k" (UID: "d723be65-afaf-4734-ada6-f8bf362ddf10") : secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:31.618852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:31.618773    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k" podStartSLOduration=1.6187589839999998 podStartE2EDuration="1.618758984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:31.617372038 +0000 UTC m=+3724.583705382" watchObservedRunningTime="2026-10-31 04:09:31.618758984 +0000 UTC m=+3724.585092331"
Oct 31 04:09:31.857801 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:31.857698    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:32.063544 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:32.063504    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:32.109008 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:32.108850    4900 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3070c2924b04: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:32.109008 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:32.108876    4900 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:32.109008 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:32.108927    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials podName:d723be65-afaf-4734-ada6-f8bf362ddf10 nodeName:}" failed. No retries permitted until 2026-10-31 04:09:33.108912384 +0000 UTC m=+3726.075245710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials") pod "payload-processing-external-model-6b9b48ccc9-qt58k" (UID: "d723be65-afaf-4734-ada6-f8bf362ddf10") : secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:32.416008 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:32.415919    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lxq9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 31 04:09:32.416008 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:32.415975    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" podUID="d7f8ce75-4372-4c11-aea6-6922ed91ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 31 04:09:32.499671 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:32.499635    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:33.122265 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:33.122236    4900 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3070c2924b04: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:33.122265 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:33.122259    4900 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:33.122519 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:33.122311    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials podName:d723be65-afaf-4734-ada6-f8bf362ddf10 nodeName:}" failed. No retries permitted until 2026-10-31 04:09:35.122296954 +0000 UTC m=+3728.088630292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials") pod "payload-processing-external-model-6b9b48ccc9-qt58k" (UID: "d723be65-afaf-4734-ada6-f8bf362ddf10") : secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:33.202164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:33.202123    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"]
Oct 31 04:09:33.416613 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:33.416530    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lxq9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 31 04:09:33.416613 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:33.416583    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" podUID="d7f8ce75-4372-4c11-aea6-6922ed91ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 31 04:09:33.448764 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:33.448708    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:33.629181 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:33.629140    4900 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Oct 31 04:09:33.629658 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:33.629240    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume podName:d723be65-afaf-4734-ada6-f8bf362ddf10 nodeName:}" failed. No retries permitted until 2026-10-31 04:09:34.129219748 +0000 UTC m=+3727.095553085 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume") pod "payload-processing-external-model-6b9b48ccc9-qt58k" (UID: "d723be65-afaf-4734-ada6-f8bf362ddf10") : configmap "payload-processing-external-model-plugins" not found
Oct 31 04:09:34.152030 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:34.151994    4900 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Oct 31 04:09:34.152316 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:34.152304    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume podName:d723be65-afaf-4734-ada6-f8bf362ddf10 nodeName:}" failed. No retries permitted until 2026-10-31 04:09:35.152282867 +0000 UTC m=+3728.118616204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume") pod "payload-processing-external-model-6b9b48ccc9-qt58k" (UID: "d723be65-afaf-4734-ada6-f8bf362ddf10") : configmap "payload-processing-external-model-plugins" not found
Oct 31 04:09:34.171646 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:34.171606    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:34.416249 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:34.416173    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lxq9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 31 04:09:34.416484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:34.416458    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" podUID="d7f8ce75-4372-4c11-aea6-6922ed91ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 31 04:09:34.454980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:34.454924    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:35.122291 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:35.122259    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:35.180014 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:35.179986    4900 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Oct 31 04:09:35.180014 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:35.180004    4900 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3070c2924b04: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:35.180014 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:35.180023    4900 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:35.180309 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:35.180058    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume podName:d723be65-afaf-4734-ada6-f8bf362ddf10 nodeName:}" failed. No retries permitted until 2026-10-31 04:09:37.180042715 +0000 UTC m=+3730.146376038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume") pod "payload-processing-external-model-6b9b48ccc9-qt58k" (UID: "d723be65-afaf-4734-ada6-f8bf362ddf10") : configmap "payload-processing-external-model-plugins" not found
Oct 31 04:09:35.180309 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:35.180073    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials podName:d723be65-afaf-4734-ada6-f8bf362ddf10 nodeName:}" failed. No retries permitted until 2026-10-31 04:09:39.180067381 +0000 UTC m=+3732.146400702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials") pod "payload-processing-external-model-6b9b48ccc9-qt58k" (UID: "d723be65-afaf-4734-ada6-f8bf362ddf10") : secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:35.416483 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:35.416395    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lxq9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 31 04:09:35.416483 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:35.416445    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" podUID="d7f8ce75-4372-4c11-aea6-6922ed91ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 31 04:09:36.415856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:36.415820    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lxq9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 31 04:09:36.416258 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:36.415882    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" podUID="d7f8ce75-4372-4c11-aea6-6922ed91ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 31 04:09:36.987590 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:36.987555    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:09:37.207171 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:37.207135    4900 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found
Oct 31 04:09:37.207391 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:37.207222    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume podName:d723be65-afaf-4734-ada6-f8bf362ddf10 nodeName:}" failed. No retries permitted until 2026-10-31 04:09:41.207199789 +0000 UTC m=+3734.173533132 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume") pod "payload-processing-external-model-6b9b48ccc9-qt58k" (UID: "d723be65-afaf-4734-ada6-f8bf362ddf10") : configmap "payload-processing-external-model-plugins" not found
Oct 31 04:09:38.456973 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:38.456923    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lxq9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:09:38.457492 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:38.456999    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" podUID="d7f8ce75-4372-4c11-aea6-6922ed91ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:09:39.233096 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:39.233050    4900 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-3070c2924b04: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:39.233096 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:39.233090    4900 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k: secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:39.233404 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:39.233155    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials podName:d723be65-afaf-4734-ada6-f8bf362ddf10 nodeName:}" failed. No retries permitted until 2026-10-31 04:09:47.233139663 +0000 UTC m=+3740.199473001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials") pod "payload-processing-external-model-6b9b48ccc9-qt58k" (UID: "d723be65-afaf-4734-ada6-f8bf362ddf10") : secret "agc-cred-3070c2924b04" not found
Oct 31 04:09:39.488545 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.488437    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lxq9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:09:39.488545 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.488514    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" podUID="d7f8ce75-4372-4c11-aea6-6922ed91ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:09:39.512170 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.512131    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k" podUID="d723be65-afaf-4734-ada6-f8bf362ddf10" containerName="payload-processing" containerID="cri-o://0bc607213db5a3ad366b58c834786cb62cb4830d27b32bf4c050ae5f0cd3750a" gracePeriod=25
Oct 31 04:09:39.565682 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.564496    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:39.565682 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.565448    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:39.566512 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.566490    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w"
Oct 31 04:09:39.677228 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.677192    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:39.790728 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.790703    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:39.841488 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.841453    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials\") pod \"d723be65-afaf-4734-ada6-f8bf362ddf10\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") "
Oct 31 04:09:39.841781 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.841514    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-routing-overlay-volume\") pod \"d723be65-afaf-4734-ada6-f8bf362ddf10\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") "
Oct 31 04:09:39.841781 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.841557    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-kube-api-access-8hdmv\" (UniqueName: \"kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-kube-api-access-8hdmv\") pod \"d723be65-afaf-4734-ada6-f8bf362ddf10\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") "
Oct 31 04:09:39.841781 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.841623    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume\") pod \"d723be65-afaf-4734-ada6-f8bf362ddf10\" (UID: \"d723be65-afaf-4734-ada6-f8bf362ddf10\") "
Oct 31 04:09:39.842567 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.842535    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-routing-overlay-volume" pod "d723be65-afaf-4734-ada6-f8bf362ddf10" (UID: "d723be65-afaf-4734-ada6-f8bf362ddf10"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:39.842701 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.842561    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume" pod "d723be65-afaf-4734-ada6-f8bf362ddf10" (UID: "d723be65-afaf-4734-ada6-f8bf362ddf10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:39.848718 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.845159    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials" pod "d723be65-afaf-4734-ada6-f8bf362ddf10" (UID: "d723be65-afaf-4734-ada6-f8bf362ddf10"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:39.853555 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.852603    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-kube-api-access-8hdmv" pod "d723be65-afaf-4734-ada6-f8bf362ddf10" (UID: "d723be65-afaf-4734-ada6-f8bf362ddf10"). InnerVolumeSpecName "kube-api-access-8hdmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:39.944129 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.944095    4900 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-provider-credentials\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:39.944129 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.944122    4900 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-routing-overlay-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:39.944331 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.944140    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hdmv\" (UniqueName: \"kubernetes.io/projected/d723be65-afaf-4734-ada6-f8bf362ddf10-kube-api-access-8hdmv\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:39.944331 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:39.944150    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d723be65-afaf-4734-ada6-f8bf362ddf10-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:40.472900 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.472870    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:40.548780 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.548747    4900 generic.go:356] "Generic (PLEG): container finished" podID="d723be65-afaf-4734-ada6-f8bf362ddf10" containerID="0bc607213db5a3ad366b58c834786cb62cb4830d27b32bf4c050ae5f0cd3750a" exitCode=0
Oct 31 04:09:40.549216 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.548827    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k" event={"ID":"d723be65-afaf-4734-ada6-f8bf362ddf10","Type":"ContainerDied","Data":"0bc607213db5a3ad366b58c834786cb62cb4830d27b32bf4c050ae5f0cd3750a"}
Oct 31 04:09:40.549216 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.548843    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"
Oct 31 04:09:40.549216 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.548885    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k" event={"ID":"d723be65-afaf-4734-ada6-f8bf362ddf10","Type":"ContainerDied","Data":"a9a5feb61981297448a3d2b9168b0967337258d33c4555906f938ed2681d6971"}
Oct 31 04:09:40.549216 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.548912    4900 scope.go:122] "RemoveContainer" containerID="0bc607213db5a3ad366b58c834786cb62cb4830d27b32bf4c050ae5f0cd3750a"
Oct 31 04:09:40.558759 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.558736    4900 scope.go:122] "RemoveContainer" containerID="0bc607213db5a3ad366b58c834786cb62cb4830d27b32bf4c050ae5f0cd3750a"
Oct 31 04:09:40.559050 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:40.559029    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc607213db5a3ad366b58c834786cb62cb4830d27b32bf4c050ae5f0cd3750a\": container with ID starting with 0bc607213db5a3ad366b58c834786cb62cb4830d27b32bf4c050ae5f0cd3750a not found: ID does not exist" containerID="0bc607213db5a3ad366b58c834786cb62cb4830d27b32bf4c050ae5f0cd3750a"
Oct 31 04:09:40.559102 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.559060    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc607213db5a3ad366b58c834786cb62cb4830d27b32bf4c050ae5f0cd3750a"} err="failed to get container status \"0bc607213db5a3ad366b58c834786cb62cb4830d27b32bf4c050ae5f0cd3750a\": rpc error: code = NotFound desc = could not find container \"0bc607213db5a3ad366b58c834786cb62cb4830d27b32bf4c050ae5f0cd3750a\": container with ID starting with 0bc607213db5a3ad366b58c834786cb62cb4830d27b32bf4c050ae5f0cd3750a not found: ID does not exist"
Oct 31 04:09:40.713080 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.713046    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"]
Oct 31 04:09:40.764535 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.764495    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b9b48ccc9-qt58k"]
Oct 31 04:09:40.809513 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.809483    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"]
Oct 31 04:09:40.810110 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.810094    4900 cpu_manager.go:414] "containerMap: removing container" podUID="d723be65-afaf-4734-ada6-f8bf362ddf10" containerName="payload-processing"
Oct 31 04:09:40.810172 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.810111    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d723be65-afaf-4734-ada6-f8bf362ddf10" containerName="payload-processing"
Oct 31 04:09:40.810215 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.810197    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="d723be65-afaf-4734-ada6-f8bf362ddf10" containerName="payload-processing"
Oct 31 04:09:40.815100 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.815078    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:40.855655 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.855609    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxjtg\" (UniqueName: \"kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-kube-api-access-fxjtg\") pod \"payload-processing-external-model-7464ffdbfc-fmsph\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") " pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:40.855852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.855668    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-routing-overlay-volume\") pod \"payload-processing-external-model-7464ffdbfc-fmsph\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") " pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:40.855852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.855830    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-provider-credentials\") pod \"payload-processing-external-model-7464ffdbfc-fmsph\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") " pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:40.855949 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.855922    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-plugins-config-volume\") pod \"payload-processing-external-model-7464ffdbfc-fmsph\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") " pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:40.868356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.868330    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7867d6f13f6b\""
Oct 31 04:09:40.868583 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.868331    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-fdwrh\""
Oct 31 04:09:40.869209 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.869188    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 04:09:40.869421 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.869395    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 04:09:40.869499 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.869415    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 04:09:40.869542 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.869511    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 04:09:40.871452 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.871433    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"]
Oct 31 04:09:40.890399 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.890367    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:40.957275 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.957241    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxjtg\" (UniqueName: \"kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-kube-api-access-fxjtg\") pod \"payload-processing-external-model-7464ffdbfc-fmsph\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") " pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:40.957275 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.957278    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-routing-overlay-volume\") pod \"payload-processing-external-model-7464ffdbfc-fmsph\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") " pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:40.957545 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.957335    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-provider-credentials\") pod \"payload-processing-external-model-7464ffdbfc-fmsph\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") " pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:40.957545 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.957393    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-plugins-config-volume\") pod \"payload-processing-external-model-7464ffdbfc-fmsph\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") " pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:40.957545 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:40.957506    4900 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7867d6f13f6b: secret "agc-cred-7867d6f13f6b" not found
Oct 31 04:09:40.957545 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:40.957526    4900 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph: secret "agc-cred-7867d6f13f6b" not found
Oct 31 04:09:40.957728 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:40.957588    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-provider-credentials podName:f57a11a2-d5cd-4bec-931e-27dd0f2a398f nodeName:}" failed. No retries permitted until 2026-10-31 04:09:41.45756644 +0000 UTC m=+3734.423899770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-provider-credentials") pod "payload-processing-external-model-7464ffdbfc-fmsph" (UID: "f57a11a2-d5cd-4bec-931e-27dd0f2a398f") : secret "agc-cred-7867d6f13f6b" not found
Oct 31 04:09:40.958010 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.957982    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-routing-overlay-volume\") pod \"payload-processing-external-model-7464ffdbfc-fmsph\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") " pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:40.958261 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.958240    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-plugins-config-volume\") pod \"payload-processing-external-model-7464ffdbfc-fmsph\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") " pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:40.987647 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:40.987619    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"]
Oct 31 04:09:40.988003 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:40.987983    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-fxjtg provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph" podUID="f57a11a2-d5cd-4bec-931e-27dd0f2a398f"
Oct 31 04:09:41.015480 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.015393    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxjtg\" (UniqueName: \"kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-kube-api-access-fxjtg\") pod \"payload-processing-external-model-7464ffdbfc-fmsph\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") " pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:41.105369 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.105337    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:41.153204 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.153163    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d723be65-afaf-4734-ada6-f8bf362ddf10" path="/var/lib/kubelet/pods/d723be65-afaf-4734-ada6-f8bf362ddf10/volumes"
Oct 31 04:09:41.309061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.308976    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:41.463196 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.463164    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-provider-credentials\") pod \"payload-processing-external-model-7464ffdbfc-fmsph\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") " pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:41.463392 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:41.463297    4900 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7867d6f13f6b: secret "agc-cred-7867d6f13f6b" not found
Oct 31 04:09:41.463392 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:41.463310    4900 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph: secret "agc-cred-7867d6f13f6b" not found
Oct 31 04:09:41.463392 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:41.463358    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-provider-credentials podName:f57a11a2-d5cd-4bec-931e-27dd0f2a398f nodeName:}" failed. No retries permitted until 2026-10-31 04:09:42.463346176 +0000 UTC m=+3735.429679511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-provider-credentials") pod "payload-processing-external-model-7464ffdbfc-fmsph" (UID: "f57a11a2-d5cd-4bec-931e-27dd0f2a398f") : secret "agc-cred-7867d6f13f6b" not found
Oct 31 04:09:41.556609 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.556582    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:41.565684 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.565621    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:41.664558 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.664520    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-routing-overlay-volume\") pod \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") "
Oct 31 04:09:41.664748 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.664670    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-kube-api-access-fxjtg\" (UniqueName: \"kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-kube-api-access-fxjtg\") pod \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") "
Oct 31 04:09:41.664748 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.664729    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-plugins-config-volume\") pod \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") "
Oct 31 04:09:41.665006 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.664980    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-routing-overlay-volume" pod "f57a11a2-d5cd-4bec-931e-27dd0f2a398f" (UID: "f57a11a2-d5cd-4bec-931e-27dd0f2a398f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:41.665177 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.665154    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-plugins-config-volume" pod "f57a11a2-d5cd-4bec-931e-27dd0f2a398f" (UID: "f57a11a2-d5cd-4bec-931e-27dd0f2a398f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:09:41.665301 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.665284    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:41.665374 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.665303    4900 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-routing-overlay-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:41.667007 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.666983    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-kube-api-access-fxjtg" pod "f57a11a2-d5cd-4bec-931e-27dd0f2a398f" (UID: "f57a11a2-d5cd-4bec-931e-27dd0f2a398f"). InnerVolumeSpecName "kube-api-access-fxjtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:09:41.766338 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:41.766302    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxjtg\" (UniqueName: \"kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-kube-api-access-fxjtg\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:42.475397 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:42.475357    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-provider-credentials\") pod \"payload-processing-external-model-7464ffdbfc-fmsph\" (UID: \"f57a11a2-d5cd-4bec-931e-27dd0f2a398f\") " pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:42.475585 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:42.475510    4900 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7867d6f13f6b: secret "agc-cred-7867d6f13f6b" not found
Oct 31 04:09:42.475585 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:42.475530    4900 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph: secret "agc-cred-7867d6f13f6b" not found
Oct 31 04:09:42.475666 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:09:42.475601    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-provider-credentials podName:f57a11a2-d5cd-4bec-931e-27dd0f2a398f nodeName:}" failed. No retries permitted until 2026-10-31 04:09:44.475586699 +0000 UTC m=+3737.441920020 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-provider-credentials") pod "payload-processing-external-model-7464ffdbfc-fmsph" (UID: "f57a11a2-d5cd-4bec-931e-27dd0f2a398f") : secret "agc-cred-7867d6f13f6b" not found
Oct 31 04:09:42.562446 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:42.562411    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"
Oct 31 04:09:42.787114 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:42.787078    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"]
Oct 31 04:09:42.790134 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:42.790107    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7464ffdbfc-fmsph"]
Oct 31 04:09:42.880298 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:42.880263    4900 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f57a11a2-d5cd-4bec-931e-27dd0f2a398f-provider-credentials\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:09:43.153276 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:43.153198    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57a11a2-d5cd-4bec-931e-27dd0f2a398f" path="/var/lib/kubelet/pods/f57a11a2-d5cd-4bec-931e-27dd0f2a398f/volumes"
Oct 31 04:09:44.707364 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:44.707328    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:44.915102 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:44.915069    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:45.355817 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:45.333371    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:45.555009 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:45.554974    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:46.785953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:46.785916    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:49.614129 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:49.614097    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:49.839576 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:49.839541    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:50.319983 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:50.319953    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:52.076910 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:52.076870    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:52.978746 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:52.978704    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:53.192205 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.192171    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:53.289714 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.289676    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"]
Oct 31 04:09:53.293979 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.293962    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"
Oct 31 04:09:53.346929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.346897    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f0e007\""
Oct 31 04:09:53.399832 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.399798    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"]
Oct 31 04:09:53.402377 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.402348    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"]
Oct 31 04:09:53.406469 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.406447    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"
Oct 31 04:09:53.441087 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.421617    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:53.441087 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.422553    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695cf523-b73f-4e3d-999e-eb0b36ad3ed5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5\" (UID: \"695cf523-b73f-4e3d-999e-eb0b36ad3ed5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"
Oct 31 04:09:53.441087 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.422711    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r5zf\" (UniqueName: \"kubernetes.io/projected/695cf523-b73f-4e3d-999e-eb0b36ad3ed5-kube-api-access-7r5zf\") pod \"payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5\" (UID: \"695cf523-b73f-4e3d-999e-eb0b36ad3ed5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"
Oct 31 04:09:53.460097 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.460066    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f0e007-dockercfg-5wnrq\""
Oct 31 04:09:53.470216 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.470175    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"]
Oct 31 04:09:53.523931 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.523892    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695cf523-b73f-4e3d-999e-eb0b36ad3ed5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5\" (UID: \"695cf523-b73f-4e3d-999e-eb0b36ad3ed5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"
Oct 31 04:09:53.524132 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.523946    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv\" (UID: \"f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"
Oct 31 04:09:53.524132 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.524039    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r5zf\" (UniqueName: \"kubernetes.io/projected/695cf523-b73f-4e3d-999e-eb0b36ad3ed5-kube-api-access-7r5zf\") pod \"payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5\" (UID: \"695cf523-b73f-4e3d-999e-eb0b36ad3ed5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"
Oct 31 04:09:53.524286 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.524258    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6nsv\" (UniqueName: \"kubernetes.io/projected/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473-kube-api-access-t6nsv\") pod \"payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv\" (UID: \"f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"
Oct 31 04:09:53.524631 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.524613    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695cf523-b73f-4e3d-999e-eb0b36ad3ed5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5\" (UID: \"695cf523-b73f-4e3d-999e-eb0b36ad3ed5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"
Oct 31 04:09:53.581528 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.581464    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r5zf\" (UniqueName: \"kubernetes.io/projected/695cf523-b73f-4e3d-999e-eb0b36ad3ed5-kube-api-access-7r5zf\") pod \"payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5\" (UID: \"695cf523-b73f-4e3d-999e-eb0b36ad3ed5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"
Oct 31 04:09:53.607127 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.607095    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"
Oct 31 04:09:53.625739 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.625714    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv\" (UID: \"f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"
Oct 31 04:09:53.625966 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.625948    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6nsv\" (UniqueName: \"kubernetes.io/projected/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473-kube-api-access-t6nsv\") pod \"payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv\" (UID: \"f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"
Oct 31 04:09:53.626518 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.626498    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv\" (UID: \"f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"
Oct 31 04:09:53.646348 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.646313    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:53.691865 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.684879    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6nsv\" (UniqueName: \"kubernetes.io/projected/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473-kube-api-access-t6nsv\") pod \"payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv\" (UID: \"f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"
Oct 31 04:09:53.728948 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.728915    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"
Oct 31 04:09:53.977773 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:09:53.977740    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695cf523_b73f_4e3d_999e_eb0b36ad3ed5.slice/crio-b33750bbe595a1f3f07f8ff0065d3bfd494e2014b4b7736f64bb8c361474c81d WatchSource:0}: Error finding container b33750bbe595a1f3f07f8ff0065d3bfd494e2014b4b7736f64bb8c361474c81d: Status 404 returned error can't find the container with id b33750bbe595a1f3f07f8ff0065d3bfd494e2014b4b7736f64bb8c361474c81d
Oct 31 04:09:53.980350 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:53.980323    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"]
Oct 31 04:09:54.100083 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:09:54.100046    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bbb717_ecfb_4dd5_ae7a_82f80d2d6473.slice/crio-bdad99e1d997f4936f28f02e57c569cabb08572a2356e926500a830abc92e4a8 WatchSource:0}: Error finding container bdad99e1d997f4936f28f02e57c569cabb08572a2356e926500a830abc92e4a8: Status 404 returned error can't find the container with id bdad99e1d997f4936f28f02e57c569cabb08572a2356e926500a830abc92e4a8
Oct 31 04:09:54.102266 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:54.102242    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"]
Oct 31 04:09:54.633172 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:54.633127    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5" event={"ID":"695cf523-b73f-4e3d-999e-eb0b36ad3ed5","Type":"ContainerStarted","Data":"9e82864daa0f5e196936d3f0c630ba1f2c4de2a54b8af1a855dcc455ced3e15e"}
Oct 31 04:09:54.633639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:54.633197    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5" event={"ID":"695cf523-b73f-4e3d-999e-eb0b36ad3ed5","Type":"ContainerStarted","Data":"b33750bbe595a1f3f07f8ff0065d3bfd494e2014b4b7736f64bb8c361474c81d"}
Oct 31 04:09:54.633639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:54.633225    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"
Oct 31 04:09:54.634852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:54.634825    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv" event={"ID":"f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473","Type":"ContainerStarted","Data":"932d92e8c1a33cd78f1d5889eac91ac1b7474f119543f1ad2ec426594f8f990f"}
Oct 31 04:09:54.634996 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:54.634868    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv" event={"ID":"f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473","Type":"ContainerStarted","Data":"bdad99e1d997f4936f28f02e57c569cabb08572a2356e926500a830abc92e4a8"}
Oct 31 04:09:54.634996 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:54.634903    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"
Oct 31 04:09:54.749648 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:54.749593    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5" podStartSLOduration=1.749579823 podStartE2EDuration="1.749579823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:54.747158093 +0000 UTC m=+3747.713491439" watchObservedRunningTime="2026-10-31 04:09:54.749579823 +0000 UTC m=+3747.715913168"
Oct 31 04:09:54.860969 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:54.860906    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv" podStartSLOduration=1.860885495 podStartE2EDuration="1.860885495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:09:54.860765135 +0000 UTC m=+3747.827098480" watchObservedRunningTime="2026-10-31 04:09:54.860885495 +0000 UTC m=+3747.827218840"
Oct 31 04:09:54.959882 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:54.959779    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:55.448554 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:55.448519    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"]
Oct 31 04:09:55.448761 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:55.448737    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9" podUID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" containerName="istio-proxy" containerID="cri-o://89d38cb4b524be92408b5629fb387bd7246875250586feb762695f701a5e5995" gracePeriod=30
Oct 31 04:09:55.524465 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:55.524427    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:56.426003 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:56.425967    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"]
Oct 31 04:09:56.472714 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:56.472676    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:56.574294 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:56.574259    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"]
Oct 31 04:09:56.695840 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:56.695730    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:56.906700 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:56.906664    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:57.112082 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:57.112051    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:09:57.721613 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:09:57.721576    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:00.689933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.689896    4900 generic.go:356] "Generic (PLEG): container finished" podID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" containerID="89d38cb4b524be92408b5629fb387bd7246875250586feb762695f701a5e5995" exitCode=0
Oct 31 04:10:00.690355 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.690004    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9" event={"ID":"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7","Type":"ContainerDied","Data":"89d38cb4b524be92408b5629fb387bd7246875250586feb762695f701a5e5995"}
Oct 31 04:10:00.690355 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.690085    4900 scope.go:122] "RemoveContainer" containerID="8490b1f2c2943ab88196226ae0ed530ab326571f248bc85000f074e110f35777"
Oct 31 04:10:00.715130 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.715110    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:10:00.810651 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.810623    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-credential-socket\") pod \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") "
Oct 31 04:10:00.810806 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.810710    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-envoy\") pod \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") "
Oct 31 04:10:00.810806 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.810737    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-data\") pod \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") "
Oct 31 04:10:00.810806 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.810755    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-token\" (UniqueName: \"kubernetes.io/projected/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-token\") pod \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") "
Oct 31 04:10:00.810947 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.810783    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-workload-certs\") pod \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") "
Oct 31 04:10:00.810986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.810946    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-credential-socket" pod "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" (UID: "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:00.811058 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.811038    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istiod-ca-cert\") pod \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") "
Oct 31 04:10:00.811127 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.811084    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-workload-certs" pod "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" (UID: "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:00.811225 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.811193    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-podinfo\") pod \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") "
Oct 31 04:10:00.811300 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.811254    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-workload-socket\") pod \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") "
Oct 31 04:10:00.811356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.811329    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-kube-api-access-6ddz6\" (UniqueName: \"kubernetes.io/projected/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-kube-api-access-6ddz6\") pod \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\" (UID: \"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7\") "
Oct 31 04:10:00.811419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.811353    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-data" pod "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" (UID: "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:00.811419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.811389    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istiod-ca-cert" pod "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" (UID: "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:00.811589 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.811570    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-workload-socket" pod "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" (UID: "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:00.811862 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.811842    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:00.811959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.811865    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:00.811959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.811880    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:00.811959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.811894    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:00.811959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.811908    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:00.813191 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.813162    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-envoy" pod "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" (UID: "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:00.813509 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.813480    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-podinfo" pod "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" (UID: "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:10:00.813604 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.813508    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-kube-api-access-6ddz6" pod "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" (UID: "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7"). InnerVolumeSpecName "kube-api-access-6ddz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:00.813704 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.813687    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-token" pod "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" (UID: "c9fad7ef-4579-40d5-a7a2-e841c2a98bb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:00.913188 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.913154    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:00.913188 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.913189    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ddz6\" (UniqueName: \"kubernetes.io/projected/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-kube-api-access-6ddz6\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:00.913380 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.913204    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:00.913380 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:00.913217    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:01.426944 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.426854    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm" podUID="d7190e98-41c5-426a-b7a3-ded4161c3266" containerName="payload-processing" containerID="cri-o://084d2104fd5864660f49ae65a7826e0a582862aa329c68ed3940e0a51640c291" gracePeriod=25
Oct 31 04:10:01.574815 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.574752    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv" podUID="0050108d-215a-467c-8244-3c882c1b29c6" containerName="payload-pre-processing" containerID="cri-o://d035a395a96cea91cd237af128b06a69d4e3a1c04698f7b4627d0d49e4a4ee11" gracePeriod=25
Oct 31 04:10:01.699135 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.699095    4900 generic.go:356] "Generic (PLEG): container finished" podID="0050108d-215a-467c-8244-3c882c1b29c6" containerID="d035a395a96cea91cd237af128b06a69d4e3a1c04698f7b4627d0d49e4a4ee11" exitCode=0
Oct 31 04:10:01.699455 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.699156    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv" event={"ID":"0050108d-215a-467c-8244-3c882c1b29c6","Type":"ContainerDied","Data":"d035a395a96cea91cd237af128b06a69d4e3a1c04698f7b4627d0d49e4a4ee11"}
Oct 31 04:10:01.700746 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.700723    4900 generic.go:356] "Generic (PLEG): container finished" podID="d7190e98-41c5-426a-b7a3-ded4161c3266" containerID="084d2104fd5864660f49ae65a7826e0a582862aa329c68ed3940e0a51640c291" exitCode=0
Oct 31 04:10:01.700883 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.700752    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm" event={"ID":"d7190e98-41c5-426a-b7a3-ded4161c3266","Type":"ContainerDied","Data":"084d2104fd5864660f49ae65a7826e0a582862aa329c68ed3940e0a51640c291"}
Oct 31 04:10:01.700883 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.700852    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm" event={"ID":"d7190e98-41c5-426a-b7a3-ded4161c3266","Type":"ContainerDied","Data":"b227fd8410a6f97bd863042f557dce24d7ace986b0f7b75f7899049757b5c8cf"}
Oct 31 04:10:01.700996 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.700885    4900 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b227fd8410a6f97bd863042f557dce24d7ace986b0f7b75f7899049757b5c8cf"
Oct 31 04:10:01.702420 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.702396    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"
Oct 31 04:10:01.702521 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.702396    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9" event={"ID":"c9fad7ef-4579-40d5-a7a2-e841c2a98bb7","Type":"ContainerDied","Data":"97d33cb294f74b759440a6b2c1397e7e6e064393c16cec90fccc834800cd045f"}
Oct 31 04:10:01.702577 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.702528    4900 scope.go:122] "RemoveContainer" containerID="89d38cb4b524be92408b5629fb387bd7246875250586feb762695f701a5e5995"
Oct 31 04:10:01.703987 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.703970    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"
Oct 31 04:10:01.721663 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.721640    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7190e98-41c5-426a-b7a3-ded4161c3266-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7190e98-41c5-426a-b7a3-ded4161c3266-plugins-config-volume\") pod \"d7190e98-41c5-426a-b7a3-ded4161c3266\" (UID: \"d7190e98-41c5-426a-b7a3-ded4161c3266\") "
Oct 31 04:10:01.721774 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.721726    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7190e98-41c5-426a-b7a3-ded4161c3266-kube-api-access-8v66l\" (UniqueName: \"kubernetes.io/projected/d7190e98-41c5-426a-b7a3-ded4161c3266-kube-api-access-8v66l\") pod \"d7190e98-41c5-426a-b7a3-ded4161c3266\" (UID: \"d7190e98-41c5-426a-b7a3-ded4161c3266\") "
Oct 31 04:10:01.722217 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.722179    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7190e98-41c5-426a-b7a3-ded4161c3266-plugins-config-volume" pod "d7190e98-41c5-426a-b7a3-ded4161c3266" (UID: "d7190e98-41c5-426a-b7a3-ded4161c3266"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:01.723739 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.723717    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7190e98-41c5-426a-b7a3-ded4161c3266-kube-api-access-8v66l" pod "d7190e98-41c5-426a-b7a3-ded4161c3266" (UID: "d7190e98-41c5-426a-b7a3-ded4161c3266"). InnerVolumeSpecName "kube-api-access-8v66l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:01.821939 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.821914    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"
Oct 31 04:10:01.823119 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.823096    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7190e98-41c5-426a-b7a3-ded4161c3266-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:01.823119 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.823117    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v66l\" (UniqueName: \"kubernetes.io/projected/d7190e98-41c5-426a-b7a3-ded4161c3266-kube-api-access-8v66l\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:01.924528 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.924499    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0050108d-215a-467c-8244-3c882c1b29c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0050108d-215a-467c-8244-3c882c1b29c6-plugins-config-volume\") pod \"0050108d-215a-467c-8244-3c882c1b29c6\" (UID: \"0050108d-215a-467c-8244-3c882c1b29c6\") "
Oct 31 04:10:01.924714 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.924541    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0050108d-215a-467c-8244-3c882c1b29c6-kube-api-access-8g529\" (UniqueName: \"kubernetes.io/projected/0050108d-215a-467c-8244-3c882c1b29c6-kube-api-access-8g529\") pod \"0050108d-215a-467c-8244-3c882c1b29c6\" (UID: \"0050108d-215a-467c-8244-3c882c1b29c6\") "
Oct 31 04:10:01.924971 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.924942    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0050108d-215a-467c-8244-3c882c1b29c6-plugins-config-volume" pod "0050108d-215a-467c-8244-3c882c1b29c6" (UID: "0050108d-215a-467c-8244-3c882c1b29c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:01.925094 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.925042    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0050108d-215a-467c-8244-3c882c1b29c6-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:01.926708 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:01.926692    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0050108d-215a-467c-8244-3c882c1b29c6-kube-api-access-8g529" pod "0050108d-215a-467c-8244-3c882c1b29c6" (UID: "0050108d-215a-467c-8244-3c882c1b29c6"). InnerVolumeSpecName "kube-api-access-8g529". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:02.025718 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:02.025688    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g529\" (UniqueName: \"kubernetes.io/projected/0050108d-215a-467c-8244-3c882c1b29c6-kube-api-access-8g529\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:02.096306 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:02.096271    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"]
Oct 31 04:10:02.102243 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:02.102218    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-e8fa21-openshift-default-5b44854b5c-cvjc9"]
Oct 31 04:10:02.267481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:02.267450    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:02.663647 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:02.663614    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"]
Oct 31 04:10:02.722827 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:02.721930    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"
Oct 31 04:10:02.722827 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:02.721954    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv" event={"ID":"0050108d-215a-467c-8244-3c882c1b29c6","Type":"ContainerDied","Data":"d74bc94df3b3dcb54cebb3afc1eabb1a7c79356200d23972fc966edbf03ffb7f"}
Oct 31 04:10:02.722827 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:02.722022    4900 scope.go:122] "RemoveContainer" containerID="d035a395a96cea91cd237af128b06a69d4e3a1c04698f7b4627d0d49e4a4ee11"
Oct 31 04:10:02.725064 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:02.725039    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"
Oct 31 04:10:02.779536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:02.779451    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:02.892877 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:02.892841    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"]
Oct 31 04:10:02.895219 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:02.895199    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d253a5-fb684cb79-vt6qv"]
Oct 31 04:10:02.995626 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:02.995595    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:03.005075 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:03.005051    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"]
Oct 31 04:10:03.007458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:03.007432    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d253a5-846cf546b7-22cbm"]
Oct 31 04:10:03.161856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:03.158099    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0050108d-215a-467c-8244-3c882c1b29c6" path="/var/lib/kubelet/pods/0050108d-215a-467c-8244-3c882c1b29c6/volumes"
Oct 31 04:10:03.161856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:03.158832    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" path="/var/lib/kubelet/pods/c9fad7ef-4579-40d5-a7a2-e841c2a98bb7/volumes"
Oct 31 04:10:03.161856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:03.159406    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7190e98-41c5-426a-b7a3-ded4161c3266" path="/var/lib/kubelet/pods/d7190e98-41c5-426a-b7a3-ded4161c3266/volumes"
Oct 31 04:10:03.574252 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:03.574223    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:04.367023 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:04.366986    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:05.651471 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:05.651439    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"
Oct 31 04:10:05.651854 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:05.651508    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"
Oct 31 04:10:06.984883 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:10:06.984845    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:10:07.664088 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:07.664020    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg" podUID="0280841e-0df8-4917-bde3-e88f014d1151" containerName="payload-pre-processing" containerID="cri-o://9606c3c6f9e6bc45e7ca24502decc845776f724eedb9601c81d9c58ec70f4645" gracePeriod=25
Oct 31 04:10:07.928726 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:07.928655    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"
Oct 31 04:10:07.996613 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:07.996585    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0280841e-0df8-4917-bde3-e88f014d1151-kube-api-access-bs8xf\" (UniqueName: \"kubernetes.io/projected/0280841e-0df8-4917-bde3-e88f014d1151-kube-api-access-bs8xf\") pod \"0280841e-0df8-4917-bde3-e88f014d1151\" (UID: \"0280841e-0df8-4917-bde3-e88f014d1151\") "
Oct 31 04:10:07.997090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:07.996678    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0280841e-0df8-4917-bde3-e88f014d1151-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0280841e-0df8-4917-bde3-e88f014d1151-plugins-config-volume\") pod \"0280841e-0df8-4917-bde3-e88f014d1151\" (UID: \"0280841e-0df8-4917-bde3-e88f014d1151\") "
Oct 31 04:10:07.997090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:07.997018    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0280841e-0df8-4917-bde3-e88f014d1151-plugins-config-volume" pod "0280841e-0df8-4917-bde3-e88f014d1151" (UID: "0280841e-0df8-4917-bde3-e88f014d1151"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:07.997171 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:07.997155    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0280841e-0df8-4917-bde3-e88f014d1151-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:07.998681 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:07.998660    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0280841e-0df8-4917-bde3-e88f014d1151-kube-api-access-bs8xf" pod "0280841e-0df8-4917-bde3-e88f014d1151" (UID: "0280841e-0df8-4917-bde3-e88f014d1151"). InnerVolumeSpecName "kube-api-access-bs8xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:08.098154 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:08.098125    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs8xf\" (UniqueName: \"kubernetes.io/projected/0280841e-0df8-4917-bde3-e88f014d1151-kube-api-access-bs8xf\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:08.753466 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:08.753430    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:08.769091 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:08.769060    4900 generic.go:356] "Generic (PLEG): container finished" podID="0280841e-0df8-4917-bde3-e88f014d1151" containerID="9606c3c6f9e6bc45e7ca24502decc845776f724eedb9601c81d9c58ec70f4645" exitCode=0
Oct 31 04:10:08.769290 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:08.769140    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg" event={"ID":"0280841e-0df8-4917-bde3-e88f014d1151","Type":"ContainerDied","Data":"9606c3c6f9e6bc45e7ca24502decc845776f724eedb9601c81d9c58ec70f4645"}
Oct 31 04:10:08.769290 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:08.769198    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"
Oct 31 04:10:08.769468 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:08.769446    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg" event={"ID":"0280841e-0df8-4917-bde3-e88f014d1151","Type":"ContainerDied","Data":"b16693dc065acdf0017bd6a84c3bc7b793a2b1cfe64647a80e98b4459a5f388b"}
Oct 31 04:10:08.769516 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:08.769485    4900 scope.go:122] "RemoveContainer" containerID="9606c3c6f9e6bc45e7ca24502decc845776f724eedb9601c81d9c58ec70f4645"
Oct 31 04:10:08.779195 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:08.779179    4900 scope.go:122] "RemoveContainer" containerID="9606c3c6f9e6bc45e7ca24502decc845776f724eedb9601c81d9c58ec70f4645"
Oct 31 04:10:08.779488 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:10:08.779468    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9606c3c6f9e6bc45e7ca24502decc845776f724eedb9601c81d9c58ec70f4645\": container with ID starting with 9606c3c6f9e6bc45e7ca24502decc845776f724eedb9601c81d9c58ec70f4645 not found: ID does not exist" containerID="9606c3c6f9e6bc45e7ca24502decc845776f724eedb9601c81d9c58ec70f4645"
Oct 31 04:10:08.779546 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:08.779495    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9606c3c6f9e6bc45e7ca24502decc845776f724eedb9601c81d9c58ec70f4645"} err="failed to get container status \"9606c3c6f9e6bc45e7ca24502decc845776f724eedb9601c81d9c58ec70f4645\": rpc error: code = NotFound desc = could not find container \"9606c3c6f9e6bc45e7ca24502decc845776f724eedb9601c81d9c58ec70f4645\": container with ID starting with 9606c3c6f9e6bc45e7ca24502decc845776f724eedb9601c81d9c58ec70f4645 not found: ID does not exist"
Oct 31 04:10:08.958013 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:08.957984    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"]
Oct 31 04:10:09.007698 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:09.007625    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0d5219-54d6bbdc55-zszdg"]
Oct 31 04:10:09.153179 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:09.153146    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0280841e-0df8-4917-bde3-e88f014d1151" path="/var/lib/kubelet/pods/0280841e-0df8-4917-bde3-e88f014d1151/volumes"
Oct 31 04:10:09.485247 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:09.485215    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:13.441519 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.441483    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"]
Oct 31 04:10:13.442077 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.442061    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0280841e-0df8-4917-bde3-e88f014d1151" containerName="payload-pre-processing"
Oct 31 04:10:13.442126 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.442078    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0280841e-0df8-4917-bde3-e88f014d1151" containerName="payload-pre-processing"
Oct 31 04:10:13.442126 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.442091    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" containerName="istio-proxy"
Oct 31 04:10:13.442126 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.442097    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" containerName="istio-proxy"
Oct 31 04:10:13.442126 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.442115    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" containerName="istio-proxy"
Oct 31 04:10:13.442126 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.442120    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" containerName="istio-proxy"
Oct 31 04:10:13.442126 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.442129    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0050108d-215a-467c-8244-3c882c1b29c6" containerName="payload-pre-processing"
Oct 31 04:10:13.442315 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.442133    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0050108d-215a-467c-8244-3c882c1b29c6" containerName="payload-pre-processing"
Oct 31 04:10:13.442315 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.442140    4900 cpu_manager.go:414] "containerMap: removing container" podUID="d7190e98-41c5-426a-b7a3-ded4161c3266" containerName="payload-processing"
Oct 31 04:10:13.442315 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.442145    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7190e98-41c5-426a-b7a3-ded4161c3266" containerName="payload-processing"
Oct 31 04:10:13.442315 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.442225    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" containerName="istio-proxy"
Oct 31 04:10:13.442315 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.442232    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0050108d-215a-467c-8244-3c882c1b29c6" containerName="payload-pre-processing"
Oct 31 04:10:13.442315 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.442239    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7190e98-41c5-426a-b7a3-ded4161c3266" containerName="payload-processing"
Oct 31 04:10:13.442315 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.442250    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0280841e-0df8-4917-bde3-e88f014d1151" containerName="payload-pre-processing"
Oct 31 04:10:13.446976 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.446958    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.553546 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.553514    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"]
Oct 31 04:10:13.562780 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.562748    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.562968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.562831    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191f5b84-f796-4f2b-8cba-a8948506b26b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.563011 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.562953    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.563093 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.563075    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.563188 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.563166    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.563257 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.563238    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.563342 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.563327    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjj7v\" (UniqueName: \"kubernetes.io/projected/191f5b84-f796-4f2b-8cba-a8948506b26b-kube-api-access-mjj7v\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.563401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.563372    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.563450 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.563399    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.664567 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.664530    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjj7v\" (UniqueName: \"kubernetes.io/projected/191f5b84-f796-4f2b-8cba-a8948506b26b-kube-api-access-mjj7v\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.664567 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.664570    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.664838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.664594    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.664838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.664658    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.664838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.664689    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191f5b84-f796-4f2b-8cba-a8948506b26b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.664838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.664718    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.664838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.664777    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.665131 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.664842    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.665131 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.664872    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.665131 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.665040    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.665131 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.665109    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.665289 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.665218    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.665344 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.665313    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.665502 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.665480    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191f5b84-f796-4f2b-8cba-a8948506b26b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.667036 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.667020    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.667211 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.667196    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.722033 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.721960    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjj7v\" (UniqueName: \"kubernetes.io/projected/191f5b84-f796-4f2b-8cba-a8948506b26b-kube-api-access-mjj7v\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.722241 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.722221    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-gh4g9\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:13.760227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:13.760191    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:14.180848 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:10:14.180816    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191f5b84_f796_4f2b_8cba_a8948506b26b.slice/crio-b892cc077d2e45aaae4d1feed4135d40066450dda979f46472f1a5c10952b85e WatchSource:0}: Error finding container b892cc077d2e45aaae4d1feed4135d40066450dda979f46472f1a5c10952b85e: Status 404 returned error can't find the container with id b892cc077d2e45aaae4d1feed4135d40066450dda979f46472f1a5c10952b85e
Oct 31 04:10:14.183493 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:14.183458    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:10:14.183600 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:14.183487    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"]
Oct 31 04:10:14.183600 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:14.183523    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:10:14.183600 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:14.183553    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:10:14.587228 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:14.587195    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:14.806999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:14.806960    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:14.808166 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:14.808142    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9" event={"ID":"191f5b84-f796-4f2b-8cba-a8948506b26b","Type":"ContainerStarted","Data":"e80111bd79a1f16a5709f13fa5fbfb8c7df78e325cffe59edc5682cc526e81fc"}
Oct 31 04:10:14.808284 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:14.808195    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9" event={"ID":"191f5b84-f796-4f2b-8cba-a8948506b26b","Type":"ContainerStarted","Data":"b892cc077d2e45aaae4d1feed4135d40066450dda979f46472f1a5c10952b85e"}
Oct 31 04:10:14.926207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:14.926096    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9" podStartSLOduration=1.926076632 podStartE2EDuration="1.926076632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:14.925997899 +0000 UTC m=+3767.892331244" watchObservedRunningTime="2026-10-31 04:10:14.926076632 +0000 UTC m=+3767.892409977"
Oct 31 04:10:15.760594 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:15.760548    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:15.762410 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:15.762383    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gh4g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 04:10:15.762558 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:15.762458    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9" podUID="191f5b84-f796-4f2b-8cba-a8948506b26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 04:10:16.296830 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:16.296759    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:16.766071 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:16.766033    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gh4g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 04:10:16.766676 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:16.766642    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9" podUID="191f5b84-f796-4f2b-8cba-a8948506b26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 04:10:16.814472 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:16.814440    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"]
Oct 31 04:10:16.814700 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:16.814670    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p" podUID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerName="istio-proxy" containerID="cri-o://ff1bbfede1b3f17587662b373e8376832e92363b1479c6062245251f071b4084" gracePeriod=30
Oct 31 04:10:16.891370 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:16.891337    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:17.761303 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:17.761266    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gh4g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 04:10:17.761475 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:17.761338    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9" podUID="191f5b84-f796-4f2b-8cba-a8948506b26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 04:10:17.929330 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:17.929297    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:18.484461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.484428    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"]
Oct 31 04:10:18.485150 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.485133    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9fad7ef-4579-40d5-a7a2-e841c2a98bb7" containerName="istio-proxy"
Oct 31 04:10:18.488679 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.488664    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.592106 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.592073    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"]
Oct 31 04:10:18.605998 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.605958    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-83f052-openshift-default-dockercfg-bbkcj\""
Oct 31 04:10:18.632072 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.632038    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-data\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.632364 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.632328    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-token\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.632611 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.632581    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-workload-socket\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.632852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.632832    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6xjq\" (UniqueName: \"kubernetes.io/projected/7af499b8-dbd8-4a13-b009-af4aff66fc7b-kube-api-access-b6xjq\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.633023 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.632999    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-envoy\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.633228 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.633214    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istiod-ca-cert\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.633329 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.633314    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-workload-certs\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.633405 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.633392    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-podinfo\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.633486 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.633473    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-credential-socket\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.667501 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.667475    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:18.735062 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.734963    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istiod-ca-cert\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.735918 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.735891    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-workload-certs\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.736245 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.736224    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-workload-certs\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.736494 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.736477    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-podinfo\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.737227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.737208    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-credential-socket\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.737898 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.737673    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-credential-socket\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.738388 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.738368    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-data\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.739039 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.739021    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-token\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.739872 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.739853    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-workload-socket\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.740327 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.740306    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6xjq\" (UniqueName: \"kubernetes.io/projected/7af499b8-dbd8-4a13-b009-af4aff66fc7b-kube-api-access-b6xjq\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.740779 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.740756    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-envoy\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.741394 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.738811    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-data\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.741552 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.740210    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-workload-socket\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.741666 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.739780    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istiod-ca-cert\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.763587 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.763182    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gh4g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 04:10:18.763587 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.763233    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9" podUID="191f5b84-f796-4f2b-8cba-a8948506b26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 04:10:18.765718 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.765684    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-podinfo\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.767639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.767614    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-envoy\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.799259 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.799236    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-token\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.799834 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.799813    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6xjq\" (UniqueName: \"kubernetes.io/projected/7af499b8-dbd8-4a13-b009-af4aff66fc7b-kube-api-access-b6xjq\") pod \"e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") " pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.803640 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.803369    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:18.893244 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:18.893184    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:19.261138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:19.261096    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"]
Oct 31 04:10:19.266450 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:10:19.266425    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af499b8_dbd8_4a13_b009_af4aff66fc7b.slice/crio-23076b2b18fba5ade6f781e41d65c8857e8892563ac6f27cde7951ad3c142f91 WatchSource:0}: Error finding container 23076b2b18fba5ade6f781e41d65c8857e8892563ac6f27cde7951ad3c142f91: Status 404 returned error can't find the container with id 23076b2b18fba5ade6f781e41d65c8857e8892563ac6f27cde7951ad3c142f91
Oct 31 04:10:19.268855 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:19.268816    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:10:19.268951 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:19.268901    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:10:19.268951 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:19.268941    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:10:19.761809 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:19.761137    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gh4g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 04:10:19.761809 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:19.761196    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9" podUID="191f5b84-f796-4f2b-8cba-a8948506b26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 04:10:19.809002 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:19.808968    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:19.865559 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:19.865516    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm" event={"ID":"7af499b8-dbd8-4a13-b009-af4aff66fc7b","Type":"ContainerStarted","Data":"d9966e8ef70e83ce0d49a439548259d433ffdde2d3e511c4de1cf6c65be1849f"}
Oct 31 04:10:19.865741 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:19.865577    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm" event={"ID":"7af499b8-dbd8-4a13-b009-af4aff66fc7b","Type":"ContainerStarted","Data":"23076b2b18fba5ade6f781e41d65c8857e8892563ac6f27cde7951ad3c142f91"}
Oct 31 04:10:19.982670 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:19.982619    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm" podStartSLOduration=1.98260544 podStartE2EDuration="1.98260544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:19.980734549 +0000 UTC m=+3772.947067894" watchObservedRunningTime="2026-10-31 04:10:19.98260544 +0000 UTC m=+3772.948938861"
Oct 31 04:10:20.761334 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:20.761297    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gh4g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 04:10:20.761758 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:20.761352    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9" podUID="191f5b84-f796-4f2b-8cba-a8948506b26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 04:10:20.803852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:20.803821    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:20.809759 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:20.809732    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:20.871855 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:20.871824    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:20.872769 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:20.872748    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:10:21.193991 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:21.193908    4900 patch_prober.go:29] interesting pod/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:10:21.194164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:21.193985    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p" podUID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:10:21.761461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:21.761423    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gh4g9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 31 04:10:21.761986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:21.761480    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9" podUID="191f5b84-f796-4f2b-8cba-a8948506b26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 31 04:10:22.114511 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.114482    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:10:22.311499 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.311467    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-workload-certs\") pod \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") "
Oct 31 04:10:22.311778 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.311521    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-podinfo\") pod \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") "
Oct 31 04:10:22.311778 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.311587    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istiod-ca-cert\") pod \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") "
Oct 31 04:10:22.311952 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.311774    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-workload-certs" pod "bc1a6b6b-c18f-40f6-b408-0090cafe29e1" (UID: "bc1a6b6b-c18f-40f6-b408-0090cafe29e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:22.311952 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.311816    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-credential-socket\") pod \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") "
Oct 31 04:10:22.311952 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.311872    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-token\" (UniqueName: \"kubernetes.io/projected/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-token\") pod \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") "
Oct 31 04:10:22.311952 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.311923    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-envoy\") pod \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") "
Oct 31 04:10:22.312194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.311977    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-data\") pod \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") "
Oct 31 04:10:22.312194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.312021    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-kube-api-access-qqjxm\" (UniqueName: \"kubernetes.io/projected/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-kube-api-access-qqjxm\") pod \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") "
Oct 31 04:10:22.312194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.312087    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-workload-socket\") pod \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\" (UID: \"bc1a6b6b-c18f-40f6-b408-0090cafe29e1\") "
Oct 31 04:10:22.312194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.312157    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istiod-ca-cert" pod "bc1a6b6b-c18f-40f6-b408-0090cafe29e1" (UID: "bc1a6b6b-c18f-40f6-b408-0090cafe29e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:22.312431 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.312335    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-credential-socket" pod "bc1a6b6b-c18f-40f6-b408-0090cafe29e1" (UID: "bc1a6b6b-c18f-40f6-b408-0090cafe29e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:22.312494 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.312441    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-data" pod "bc1a6b6b-c18f-40f6-b408-0090cafe29e1" (UID: "bc1a6b6b-c18f-40f6-b408-0090cafe29e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:22.312650 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.312628    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-workload-socket" pod "bc1a6b6b-c18f-40f6-b408-0090cafe29e1" (UID: "bc1a6b6b-c18f-40f6-b408-0090cafe29e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:22.312965 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.312857    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:22.312965 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.312884    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:22.312965 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.312908    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:22.312965 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.312924    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:22.312965 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.312938    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:22.314963 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.314929    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-podinfo" pod "bc1a6b6b-c18f-40f6-b408-0090cafe29e1" (UID: "bc1a6b6b-c18f-40f6-b408-0090cafe29e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:10:22.315061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.314955    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-envoy" pod "bc1a6b6b-c18f-40f6-b408-0090cafe29e1" (UID: "bc1a6b6b-c18f-40f6-b408-0090cafe29e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:10:22.315061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.315028    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-token" pod "bc1a6b6b-c18f-40f6-b408-0090cafe29e1" (UID: "bc1a6b6b-c18f-40f6-b408-0090cafe29e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:22.318104 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.318082    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-kube-api-access-qqjxm" pod "bc1a6b6b-c18f-40f6-b408-0090cafe29e1" (UID: "bc1a6b6b-c18f-40f6-b408-0090cafe29e1"). InnerVolumeSpecName "kube-api-access-qqjxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:22.415041 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.414932    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:22.415041 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.414981    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:22.415041 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.414997    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:22.415041 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.415012    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqjxm\" (UniqueName: \"kubernetes.io/projected/bc1a6b6b-c18f-40f6-b408-0090cafe29e1-kube-api-access-qqjxm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:22.765097 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.765067    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:22.765588 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.765499    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:22.766310 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.766293    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:10:22.886166 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.886128    4900 generic.go:356] "Generic (PLEG): container finished" podID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerID="ff1bbfede1b3f17587662b373e8376832e92363b1479c6062245251f071b4084" exitCode=0
Oct 31 04:10:22.886361 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.886194    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"
Oct 31 04:10:22.886361 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.886205    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p" event={"ID":"bc1a6b6b-c18f-40f6-b408-0090cafe29e1","Type":"ContainerDied","Data":"ff1bbfede1b3f17587662b373e8376832e92363b1479c6062245251f071b4084"}
Oct 31 04:10:22.886361 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.886261    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p" event={"ID":"bc1a6b6b-c18f-40f6-b408-0090cafe29e1","Type":"ContainerDied","Data":"d1ee3be64e1325efc77ca17989c81e0b5b469c2922c96b13a4f97c43269a4f49"}
Oct 31 04:10:22.886361 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.886284    4900 scope.go:122] "RemoveContainer" containerID="ff1bbfede1b3f17587662b373e8376832e92363b1479c6062245251f071b4084"
Oct 31 04:10:22.897134 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.897114    4900 scope.go:122] "RemoveContainer" containerID="26113adcfa27137b1c1974bbea82b46e306b09b5e0cc6d6a556d1f38123e37f2"
Oct 31 04:10:22.906408 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.906388    4900 scope.go:122] "RemoveContainer" containerID="ff1bbfede1b3f17587662b373e8376832e92363b1479c6062245251f071b4084"
Oct 31 04:10:22.906692 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:10:22.906672    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff1bbfede1b3f17587662b373e8376832e92363b1479c6062245251f071b4084\": container with ID starting with ff1bbfede1b3f17587662b373e8376832e92363b1479c6062245251f071b4084 not found: ID does not exist" containerID="ff1bbfede1b3f17587662b373e8376832e92363b1479c6062245251f071b4084"
Oct 31 04:10:22.906743 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.906702    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff1bbfede1b3f17587662b373e8376832e92363b1479c6062245251f071b4084"} err="failed to get container status \"ff1bbfede1b3f17587662b373e8376832e92363b1479c6062245251f071b4084\": rpc error: code = NotFound desc = could not find container \"ff1bbfede1b3f17587662b373e8376832e92363b1479c6062245251f071b4084\": container with ID starting with ff1bbfede1b3f17587662b373e8376832e92363b1479c6062245251f071b4084 not found: ID does not exist"
Oct 31 04:10:22.906743 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.906720    4900 scope.go:122] "RemoveContainer" containerID="26113adcfa27137b1c1974bbea82b46e306b09b5e0cc6d6a556d1f38123e37f2"
Oct 31 04:10:22.906994 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:10:22.906973    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26113adcfa27137b1c1974bbea82b46e306b09b5e0cc6d6a556d1f38123e37f2\": container with ID starting with 26113adcfa27137b1c1974bbea82b46e306b09b5e0cc6d6a556d1f38123e37f2 not found: ID does not exist" containerID="26113adcfa27137b1c1974bbea82b46e306b09b5e0cc6d6a556d1f38123e37f2"
Oct 31 04:10:22.907060 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.907003    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26113adcfa27137b1c1974bbea82b46e306b09b5e0cc6d6a556d1f38123e37f2"} err="failed to get container status \"26113adcfa27137b1c1974bbea82b46e306b09b5e0cc6d6a556d1f38123e37f2\": rpc error: code = NotFound desc = could not find container \"26113adcfa27137b1c1974bbea82b46e306b09b5e0cc6d6a556d1f38123e37f2\": container with ID starting with 26113adcfa27137b1c1974bbea82b46e306b09b5e0cc6d6a556d1f38123e37f2 not found: ID does not exist"
Oct 31 04:10:22.911059 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:22.911040    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:23.044955 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:23.044854    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"]
Oct 31 04:10:23.050464 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:23.050429    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-0d5219-openshift-default-5665c7cd49-4l24p"]
Oct 31 04:10:23.140365 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:23.140334    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:23.153417 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:23.153374    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" path="/var/lib/kubelet/pods/bc1a6b6b-c18f-40f6-b408-0090cafe29e1/volumes"
Oct 31 04:10:25.171482 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:25.171451    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:26.234424 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.234389    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:26.436414 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.436371    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:26.568513 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.568434    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"]
Oct 31 04:10:26.569006 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.568992    4900 cpu_manager.go:414] "containerMap: removing container" podUID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerName="istio-proxy"
Oct 31 04:10:26.569080 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.569008    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerName="istio-proxy"
Oct 31 04:10:26.569080 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.569025    4900 cpu_manager.go:414] "containerMap: removing container" podUID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerName="istio-proxy"
Oct 31 04:10:26.569080 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.569031    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerName="istio-proxy"
Oct 31 04:10:26.569176 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.569133    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerName="istio-proxy"
Oct 31 04:10:26.569176 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.569143    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1a6b6b-c18f-40f6-b408-0090cafe29e1" containerName="istio-proxy"
Oct 31 04:10:26.573848 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.573828    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"
Oct 31 04:10:26.623964 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.623937    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"]
Oct 31 04:10:26.665897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.665859    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h79kv\" (UniqueName: \"kubernetes.io/projected/43d4c814-2191-4ff8-9897-d5d9fd69511a-kube-api-access-h79kv\") pod \"payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf\" (UID: \"43d4c814-2191-4ff8-9897-d5d9fd69511a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"
Oct 31 04:10:26.666137 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.666117    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d4c814-2191-4ff8-9897-d5d9fd69511a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf\" (UID: \"43d4c814-2191-4ff8-9897-d5d9fd69511a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"
Oct 31 04:10:26.671057 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.671027    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-83f052\""
Oct 31 04:10:26.671196 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.671132    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-83f052-dockercfg-mbh2z\""
Oct 31 04:10:26.713445 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.713407    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"]
Oct 31 04:10:26.718646 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.718606    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"
Oct 31 04:10:26.768038 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.768003    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d4c814-2191-4ff8-9897-d5d9fd69511a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf\" (UID: \"43d4c814-2191-4ff8-9897-d5d9fd69511a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"
Oct 31 04:10:26.768223 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.768057    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bab061e-96a7-4844-a278-ce81f21d9ebf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5\" (UID: \"5bab061e-96a7-4844-a278-ce81f21d9ebf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"
Oct 31 04:10:26.768223 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.768117    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skmtn\" (UniqueName: \"kubernetes.io/projected/5bab061e-96a7-4844-a278-ce81f21d9ebf-kube-api-access-skmtn\") pod \"payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5\" (UID: \"5bab061e-96a7-4844-a278-ce81f21d9ebf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"
Oct 31 04:10:26.768339 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.768284    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h79kv\" (UniqueName: \"kubernetes.io/projected/43d4c814-2191-4ff8-9897-d5d9fd69511a-kube-api-access-h79kv\") pod \"payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf\" (UID: \"43d4c814-2191-4ff8-9897-d5d9fd69511a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"
Oct 31 04:10:26.768929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.768900    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d4c814-2191-4ff8-9897-d5d9fd69511a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf\" (UID: \"43d4c814-2191-4ff8-9897-d5d9fd69511a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"
Oct 31 04:10:26.769635 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.769613    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"]
Oct 31 04:10:26.825013 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.824953    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h79kv\" (UniqueName: \"kubernetes.io/projected/43d4c814-2191-4ff8-9897-d5d9fd69511a-kube-api-access-h79kv\") pod \"payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf\" (UID: \"43d4c814-2191-4ff8-9897-d5d9fd69511a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"
Oct 31 04:10:26.869424 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.869400    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bab061e-96a7-4844-a278-ce81f21d9ebf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5\" (UID: \"5bab061e-96a7-4844-a278-ce81f21d9ebf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"
Oct 31 04:10:26.869561 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.869446    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skmtn\" (UniqueName: \"kubernetes.io/projected/5bab061e-96a7-4844-a278-ce81f21d9ebf-kube-api-access-skmtn\") pod \"payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5\" (UID: \"5bab061e-96a7-4844-a278-ce81f21d9ebf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"
Oct 31 04:10:26.870071 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.870053    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bab061e-96a7-4844-a278-ce81f21d9ebf-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5\" (UID: \"5bab061e-96a7-4844-a278-ce81f21d9ebf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"
Oct 31 04:10:26.887522 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.887502    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"
Oct 31 04:10:26.926043 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:26.926011    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skmtn\" (UniqueName: \"kubernetes.io/projected/5bab061e-96a7-4844-a278-ce81f21d9ebf-kube-api-access-skmtn\") pod \"payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5\" (UID: \"5bab061e-96a7-4844-a278-ce81f21d9ebf\") " pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"
Oct 31 04:10:27.034353 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:27.034320    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"
Oct 31 04:10:27.261761 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:10:27.261735    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d4c814_2191_4ff8_9897_d5d9fd69511a.slice/crio-c6a255adfdde12e78fd014fd681cc12130e3db28fb23287b231cc891c05a03b9 WatchSource:0}: Error finding container c6a255adfdde12e78fd014fd681cc12130e3db28fb23287b231cc891c05a03b9: Status 404 returned error can't find the container with id c6a255adfdde12e78fd014fd681cc12130e3db28fb23287b231cc891c05a03b9
Oct 31 04:10:27.264135 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:27.264114    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"]
Oct 31 04:10:27.401872 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:10:27.400920    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bab061e_96a7_4844_a278_ce81f21d9ebf.slice/crio-14129c7ce7a0421ec5553207d6c83d1c92228c52ae6d6a36b1594347b49f4a48 WatchSource:0}: Error finding container 14129c7ce7a0421ec5553207d6c83d1c92228c52ae6d6a36b1594347b49f4a48: Status 404 returned error can't find the container with id 14129c7ce7a0421ec5553207d6c83d1c92228c52ae6d6a36b1594347b49f4a48
Oct 31 04:10:27.403740 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:27.403712    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"]
Oct 31 04:10:27.918296 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:27.918264    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5" event={"ID":"5bab061e-96a7-4844-a278-ce81f21d9ebf","Type":"ContainerStarted","Data":"71375f038b0275fc4660cfaea50772d8bf850851dc73bcf19734189322889e2a"}
Oct 31 04:10:27.918480 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:27.918315    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5" event={"ID":"5bab061e-96a7-4844-a278-ce81f21d9ebf","Type":"ContainerStarted","Data":"14129c7ce7a0421ec5553207d6c83d1c92228c52ae6d6a36b1594347b49f4a48"}
Oct 31 04:10:27.918480 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:27.918357    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"
Oct 31 04:10:27.920095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:27.920070    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf" event={"ID":"43d4c814-2191-4ff8-9897-d5d9fd69511a","Type":"ContainerStarted","Data":"38867b471fdf55f6111d896f288f53155901f61692a538516ce3592b73da5cf3"}
Oct 31 04:10:27.920216 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:27.920111    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf" event={"ID":"43d4c814-2191-4ff8-9897-d5d9fd69511a","Type":"ContainerStarted","Data":"c6a255adfdde12e78fd014fd681cc12130e3db28fb23287b231cc891c05a03b9"}
Oct 31 04:10:27.920216 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:27.920144    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"
Oct 31 04:10:28.032069 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:28.031977    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5" podStartSLOduration=2.031962066 podStartE2EDuration="2.031962066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:28.031116379 +0000 UTC m=+3780.997449723" watchObservedRunningTime="2026-10-31 04:10:28.031962066 +0000 UTC m=+3780.998295411"
Oct 31 04:10:28.144756 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:28.144701    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf" podStartSLOduration=2.144679997 podStartE2EDuration="2.144679997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:28.14300668 +0000 UTC m=+3781.109340023" watchObservedRunningTime="2026-10-31 04:10:28.144679997 +0000 UTC m=+3781.111013341"
Oct 31 04:10:29.934924 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:29.934882    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:31.897496 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:31.897428    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:32.684044 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:32.684015    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:33.276155 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:33.276110    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:34.329683 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:34.329648    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:34.979085 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:34.979046    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:35.258073 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:35.258040    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:36.482460 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.482426    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"]
Oct 31 04:10:36.483349 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.483319    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk" podUID="d05dfb40-59b0-4026-941c-6c600605d6fd" containerName="maas-api" containerID="cri-o://984a94abd479889d5da283f760816e3c22d1f1fc18aa91c66bc70270ed337d9a" gracePeriod=30
Oct 31 04:10:36.638953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.638881    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:36.840165 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.840132    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:10:36.930720 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.930681    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-maas-api-tls\") pod \"d05dfb40-59b0-4026-941c-6c600605d6fd\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") "
Oct 31 04:10:36.930933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.930841    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-metrics-certs\") pod \"d05dfb40-59b0-4026-941c-6c600605d6fd\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") "
Oct 31 04:10:36.930933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.930903    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05dfb40-59b0-4026-941c-6c600605d6fd-kube-api-access-jlsml\" (UniqueName: \"kubernetes.io/projected/d05dfb40-59b0-4026-941c-6c600605d6fd-kube-api-access-jlsml\") pod \"d05dfb40-59b0-4026-941c-6c600605d6fd\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") "
Oct 31 04:10:36.931028 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.930999    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05dfb40-59b0-4026-941c-6c600605d6fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05dfb40-59b0-4026-941c-6c600605d6fd-openshift-service-ca\") pod \"d05dfb40-59b0-4026-941c-6c600605d6fd\" (UID: \"d05dfb40-59b0-4026-941c-6c600605d6fd\") "
Oct 31 04:10:36.931336 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.931301    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05dfb40-59b0-4026-941c-6c600605d6fd-openshift-service-ca" pod "d05dfb40-59b0-4026-941c-6c600605d6fd" (UID: "d05dfb40-59b0-4026-941c-6c600605d6fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:36.931479 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.931459    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d05dfb40-59b0-4026-941c-6c600605d6fd-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:36.932952 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.932929    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-maas-api-tls" pod "d05dfb40-59b0-4026-941c-6c600605d6fd" (UID: "d05dfb40-59b0-4026-941c-6c600605d6fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:10:36.933344 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.933326    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-metrics-certs" pod "d05dfb40-59b0-4026-941c-6c600605d6fd" (UID: "d05dfb40-59b0-4026-941c-6c600605d6fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:10:36.933406 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.933369    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05dfb40-59b0-4026-941c-6c600605d6fd-kube-api-access-jlsml" pod "d05dfb40-59b0-4026-941c-6c600605d6fd" (UID: "d05dfb40-59b0-4026-941c-6c600605d6fd"). InnerVolumeSpecName "kube-api-access-jlsml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:36.989817 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.986075    4900 generic.go:356] "Generic (PLEG): container finished" podID="d05dfb40-59b0-4026-941c-6c600605d6fd" containerID="984a94abd479889d5da283f760816e3c22d1f1fc18aa91c66bc70270ed337d9a" exitCode=0
Oct 31 04:10:36.989817 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.986154    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk" event={"ID":"d05dfb40-59b0-4026-941c-6c600605d6fd","Type":"ContainerDied","Data":"984a94abd479889d5da283f760816e3c22d1f1fc18aa91c66bc70270ed337d9a"}
Oct 31 04:10:36.989817 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.986205    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk" event={"ID":"d05dfb40-59b0-4026-941c-6c600605d6fd","Type":"ContainerDied","Data":"bb9d2281b50ea3b39a34341b9b72986bf7a212cf12510164dbb7d46f5cee7402"}
Oct 31 04:10:36.989817 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.986236    4900 scope.go:122] "RemoveContainer" containerID="984a94abd479889d5da283f760816e3c22d1f1fc18aa91c66bc70270ed337d9a"
Oct 31 04:10:36.989817 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:36.986398    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"
Oct 31 04:10:37.010970 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:37.010944    4900 scope.go:122] "RemoveContainer" containerID="984a94abd479889d5da283f760816e3c22d1f1fc18aa91c66bc70270ed337d9a"
Oct 31 04:10:37.011467 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:10:37.011441    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"984a94abd479889d5da283f760816e3c22d1f1fc18aa91c66bc70270ed337d9a\": container with ID starting with 984a94abd479889d5da283f760816e3c22d1f1fc18aa91c66bc70270ed337d9a not found: ID does not exist" containerID="984a94abd479889d5da283f760816e3c22d1f1fc18aa91c66bc70270ed337d9a"
Oct 31 04:10:37.011583 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:37.011476    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"984a94abd479889d5da283f760816e3c22d1f1fc18aa91c66bc70270ed337d9a"} err="failed to get container status \"984a94abd479889d5da283f760816e3c22d1f1fc18aa91c66bc70270ed337d9a\": rpc error: code = NotFound desc = could not find container \"984a94abd479889d5da283f760816e3c22d1f1fc18aa91c66bc70270ed337d9a\": container with ID starting with 984a94abd479889d5da283f760816e3c22d1f1fc18aa91c66bc70270ed337d9a not found: ID does not exist"
Oct 31 04:10:37.012836 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:10:37.012805    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:10:37.032913 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:37.032884    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:37.032913 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:37.032910    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlsml\" (UniqueName: \"kubernetes.io/projected/d05dfb40-59b0-4026-941c-6c600605d6fd-kube-api-access-jlsml\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:37.033115 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:37.032920    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d05dfb40-59b0-4026-941c-6c600605d6fd-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:37.174769 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:37.174693    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"]
Oct 31 04:10:37.174769 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:37.174731    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5901fe-5556f7d4b-rc7tk"]
Oct 31 04:10:38.570160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:38.570126    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:38.832928 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:38.832847    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:38.928163 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:38.928136    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"
Oct 31 04:10:38.928341 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:38.928198    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"
Oct 31 04:10:39.156470 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:39.156363    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05dfb40-59b0-4026-941c-6c600605d6fd" path="/var/lib/kubelet/pods/d05dfb40-59b0-4026-941c-6c600605d6fd/volumes"
Oct 31 04:10:39.247003 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:39.246968    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:39.483612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:39.483583    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:39.696837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:39.696804    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:39.996984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:39.996946    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:40.207433 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:40.207399    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:40.667087 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:40.667052    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:40.877899 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:40.877866    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:41.646433 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:41.646401    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:42.172038 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:42.172007    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:43.085839 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.085804    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:43.658149 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.658114    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"]
Oct 31 04:10:43.658692 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.658679    4900 cpu_manager.go:414] "containerMap: removing container" podUID="d05dfb40-59b0-4026-941c-6c600605d6fd" containerName="maas-api"
Oct 31 04:10:43.658733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.658693    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05dfb40-59b0-4026-941c-6c600605d6fd" containerName="maas-api"
Oct 31 04:10:43.658811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.658802    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05dfb40-59b0-4026-941c-6c600605d6fd" containerName="maas-api"
Oct 31 04:10:43.663108 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.663090    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.708298 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.708258    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"]
Oct 31 04:10:43.712213 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.712196    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.758661 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.758629    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"]
Oct 31 04:10:43.805935 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.805900    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"]
Oct 31 04:10:43.818843 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.818810    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.818957 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.818917    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.818957 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.818949    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghrvx\" (UniqueName: \"kubernetes.io/projected/86e324aa-c2e3-4524-a3f0-2d2af88b6063-kube-api-access-ghrvx\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.819034 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.818977    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.819095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.819073    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.819182 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.819163    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f0f3ad-038c-4104-ad7c-c0354f33862a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.819225 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.819206    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f0f3ad-038c-4104-ad7c-c0354f33862a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.819260 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.819247    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.819295 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.819269    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.819295 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.819289    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.819360 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.819344    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.819413 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.819399    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.819450 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.819435    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f0f3ad-038c-4104-ad7c-c0354f33862a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.819484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.819464    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.819589 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.819568    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.819635 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.819621    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.819678 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.819665    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.819816 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.819799    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dklwn\" (UniqueName: \"kubernetes.io/projected/10f0f3ad-038c-4104-ad7c-c0354f33862a-kube-api-access-dklwn\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.834973 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.834943    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:43.920625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.920545    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.920625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.920580    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghrvx\" (UniqueName: \"kubernetes.io/projected/86e324aa-c2e3-4524-a3f0-2d2af88b6063-kube-api-access-ghrvx\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.920625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.920600    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.920625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.920624    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.920999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.920665    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f0f3ad-038c-4104-ad7c-c0354f33862a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.920999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.920699    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f0f3ad-038c-4104-ad7c-c0354f33862a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.920999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.920724    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.920999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.920750    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.920999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.920777    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.920999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.920845    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.920999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.920875    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.920999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.920918    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f0f3ad-038c-4104-ad7c-c0354f33862a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.920999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.920948    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.920999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.921001    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.921540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.921043    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.921540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.921069    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.921540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.921089    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.921540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.921089    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.921540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.921152    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dklwn\" (UniqueName: \"kubernetes.io/projected/10f0f3ad-038c-4104-ad7c-c0354f33862a-kube-api-access-dklwn\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.921540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.921223    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.921540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.921424    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.921540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.921434    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f0f3ad-038c-4104-ad7c-c0354f33862a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.922020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.921623    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.922020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.921688    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.922020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.921624    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.922020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.921903    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.922020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.921982    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.922311 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.922158    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.923326 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.923297    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f0f3ad-038c-4104-ad7c-c0354f33862a-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.923778 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.923759    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.923886 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.923823    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f0f3ad-038c-4104-ad7c-c0354f33862a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.924225 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.924208    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.978143 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.978111    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghrvx\" (UniqueName: \"kubernetes.io/projected/86e324aa-c2e3-4524-a3f0-2d2af88b6063-kube-api-access-ghrvx\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.978304 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.978193    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dklwn\" (UniqueName: \"kubernetes.io/projected/10f0f3ad-038c-4104-ad7c-c0354f33862a-kube-api-access-dklwn\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:43.978444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.978425    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-jg5sl\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:43.978517 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:43.978500    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f0f3ad-038c-4104-ad7c-c0354f33862a-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-6rp2d\" (UID: \"10f0f3ad-038c-4104-ad7c-c0354f33862a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:44.025680 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.025436    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:44.054852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.054822    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:44.261192 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.261159    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:44.276168 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.276138    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:44.477909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.477877    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"]
Oct 31 04:10:44.482997 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.482971    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"
Oct 31 04:10:44.486834 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:10:44.486808    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f0f3ad_038c_4104_ad7c_c0354f33862a.slice/crio-193c3c19a985c75691901843e0b07428032bf27f16bb0f2c1aeb5b5779033efd WatchSource:0}: Error finding container 193c3c19a985c75691901843e0b07428032bf27f16bb0f2c1aeb5b5779033efd: Status 404 returned error can't find the container with id 193c3c19a985c75691901843e0b07428032bf27f16bb0f2c1aeb5b5779033efd
Oct 31 04:10:44.489605 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.489585    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"]
Oct 31 04:10:44.489605 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.489588    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:10:44.489720 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.489645    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:10:44.489720 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.489675    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:10:44.533863 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.533836    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"]
Oct 31 04:10:44.536052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.536022    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b6731b-e6fe-4b7d-b349-18acfb435f95-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8\" (UID: \"b9b6731b-e6fe-4b7d-b349-18acfb435f95\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"
Oct 31 04:10:44.536234 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.536093    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75j79\" (UniqueName: \"kubernetes.io/projected/b9b6731b-e6fe-4b7d-b349-18acfb435f95-kube-api-access-75j79\") pod \"payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8\" (UID: \"b9b6731b-e6fe-4b7d-b349-18acfb435f95\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"
Oct 31 04:10:44.552941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.552912    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e53098-dockercfg-jkdj2\""
Oct 31 04:10:44.553083 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.553013    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e53098\""
Oct 31 04:10:44.636959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.636922    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b6731b-e6fe-4b7d-b349-18acfb435f95-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8\" (UID: \"b9b6731b-e6fe-4b7d-b349-18acfb435f95\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"
Oct 31 04:10:44.637143 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.636982    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75j79\" (UniqueName: \"kubernetes.io/projected/b9b6731b-e6fe-4b7d-b349-18acfb435f95-kube-api-access-75j79\") pod \"payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8\" (UID: \"b9b6731b-e6fe-4b7d-b349-18acfb435f95\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"
Oct 31 04:10:44.637691 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.637666    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b6731b-e6fe-4b7d-b349-18acfb435f95-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8\" (UID: \"b9b6731b-e6fe-4b7d-b349-18acfb435f95\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"
Oct 31 04:10:44.697756 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.697724    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75j79\" (UniqueName: \"kubernetes.io/projected/b9b6731b-e6fe-4b7d-b349-18acfb435f95-kube-api-access-75j79\") pod \"payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8\" (UID: \"b9b6731b-e6fe-4b7d-b349-18acfb435f95\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"
Oct 31 04:10:44.715516 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.715483    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"]
Oct 31 04:10:44.716082 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:10:44.716057    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e324aa_c2e3_4524_a3f0_2d2af88b6063.slice/crio-3c62ebb282f4feae0e14a95f92bb1c3eee5748d6dab24cf4a1aa1020a8d601d6 WatchSource:0}: Error finding container 3c62ebb282f4feae0e14a95f92bb1c3eee5748d6dab24cf4a1aa1020a8d601d6: Status 404 returned error can't find the container with id 3c62ebb282f4feae0e14a95f92bb1c3eee5748d6dab24cf4a1aa1020a8d601d6
Oct 31 04:10:44.718371 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.718339    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:10:44.718458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.718407    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:10:44.718458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.718436    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:10:44.799172 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:44.799139    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"
Oct 31 04:10:45.039095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:45.038610    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl" event={"ID":"86e324aa-c2e3-4524-a3f0-2d2af88b6063","Type":"ContainerStarted","Data":"8dc4727f150855ac72e551b486df4d8f6af8e0d3965eccbdf34c2bca42ed3275"}
Oct 31 04:10:45.039095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:45.038676    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl" event={"ID":"86e324aa-c2e3-4524-a3f0-2d2af88b6063","Type":"ContainerStarted","Data":"3c62ebb282f4feae0e14a95f92bb1c3eee5748d6dab24cf4a1aa1020a8d601d6"}
Oct 31 04:10:45.040879 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:45.040847    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d" event={"ID":"10f0f3ad-038c-4104-ad7c-c0354f33862a","Type":"ContainerStarted","Data":"dc2c0d30bc1fba8c90d39d1370e88f8aa35302bd8566dd8ca65e4e4970535b7f"}
Oct 31 04:10:45.041049 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:45.040896    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d" event={"ID":"10f0f3ad-038c-4104-ad7c-c0354f33862a","Type":"ContainerStarted","Data":"193c3c19a985c75691901843e0b07428032bf27f16bb0f2c1aeb5b5779033efd"}
Oct 31 04:10:45.169816 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:45.169007    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl" podStartSLOduration=2.168987252 podStartE2EDuration="2.168987252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:45.168477756 +0000 UTC m=+3798.134811103" watchObservedRunningTime="2026-10-31 04:10:45.168987252 +0000 UTC m=+3798.135320597"
Oct 31 04:10:45.198289 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:10:45.198256    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b6731b_e6fe_4b7d_b349_18acfb435f95.slice/crio-032441eb5dbf88b6881f5b23275cfee1f1f81123385af1ff28160b463261b787 WatchSource:0}: Error finding container 032441eb5dbf88b6881f5b23275cfee1f1f81123385af1ff28160b463261b787: Status 404 returned error can't find the container with id 032441eb5dbf88b6881f5b23275cfee1f1f81123385af1ff28160b463261b787
Oct 31 04:10:45.200550 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:45.200526    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"]
Oct 31 04:10:45.276912 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:45.276890    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:45.278453 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:45.278430    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jg5sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 04:10:45.278621 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:45.278602    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl" podUID="86e324aa-c2e3-4524-a3f0-2d2af88b6063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 04:10:45.287374 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:45.287324    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d" podStartSLOduration=2.287310689 podStartE2EDuration="2.287310689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:45.284171398 +0000 UTC m=+3798.250504742" watchObservedRunningTime="2026-10-31 04:10:45.287310689 +0000 UTC m=+3798.253644032"
Oct 31 04:10:46.026340 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:46.026297    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:46.027896 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:46.027874    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6rp2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 04:10:46.027954 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:46.027924    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d" podUID="10f0f3ad-038c-4104-ad7c-c0354f33862a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 04:10:46.093972 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:46.093644    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8" event={"ID":"b9b6731b-e6fe-4b7d-b349-18acfb435f95","Type":"ContainerStarted","Data":"83d091851d8fc2e5d0359330bbf82e6c95985ceeb7636ceff6811447f1d3b0db"}
Oct 31 04:10:46.093972 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:46.093699    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8" event={"ID":"b9b6731b-e6fe-4b7d-b349-18acfb435f95","Type":"ContainerStarted","Data":"032441eb5dbf88b6881f5b23275cfee1f1f81123385af1ff28160b463261b787"}
Oct 31 04:10:46.093972 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:46.093964    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"
Oct 31 04:10:46.211444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:46.211385    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8" podStartSLOduration=2.21137064 podStartE2EDuration="2.21137064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:10:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:10:46.206192525 +0000 UTC m=+3799.172525869" watchObservedRunningTime="2026-10-31 04:10:46.21137064 +0000 UTC m=+3799.177703984"
Oct 31 04:10:46.278870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:46.278193    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jg5sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 04:10:46.278870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:46.278259    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl" podUID="86e324aa-c2e3-4524-a3f0-2d2af88b6063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 04:10:47.026520 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:47.026483    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6rp2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 04:10:47.026973 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:47.026536    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d" podUID="10f0f3ad-038c-4104-ad7c-c0354f33862a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 04:10:47.285870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:47.280406    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jg5sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 04:10:47.285870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:47.280472    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl" podUID="86e324aa-c2e3-4524-a3f0-2d2af88b6063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 04:10:48.025863 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:48.025825    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6rp2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 04:10:48.026072 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:48.025887    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d" podUID="10f0f3ad-038c-4104-ad7c-c0354f33862a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 04:10:48.278869 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:48.278311    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jg5sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 04:10:48.278869 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:48.278394    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl" podUID="86e324aa-c2e3-4524-a3f0-2d2af88b6063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 04:10:48.527011 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:48.526974    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"]
Oct 31 04:10:48.994920 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:48.994881    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:49.025984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:49.025940    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6rp2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 04:10:49.026298 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:49.025999    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d" podUID="10f0f3ad-038c-4104-ad7c-c0354f33862a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 04:10:49.192334 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:49.192295    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:49.284867 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:49.279532    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jg5sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 04:10:49.284867 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:49.279591    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl" podUID="86e324aa-c2e3-4524-a3f0-2d2af88b6063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 04:10:50.026375 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:50.026339    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6rp2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 04:10:50.026555 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:50.026390    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d" podUID="10f0f3ad-038c-4104-ad7c-c0354f33862a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 04:10:50.279875 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:50.279186    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jg5sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 04:10:50.279875 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:50.279251    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl" podUID="86e324aa-c2e3-4524-a3f0-2d2af88b6063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 04:10:51.026469 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:51.026430    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6rp2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 04:10:51.026971 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:51.026490    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d" podUID="10f0f3ad-038c-4104-ad7c-c0354f33862a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 04:10:51.152975 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:51.152943    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:51.277012 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:51.276923    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jg5sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 04:10:51.277012 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:51.276978    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl" podUID="86e324aa-c2e3-4524-a3f0-2d2af88b6063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 04:10:51.475800 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:51.475766    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:51.682393 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:51.682293    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:52.026734 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:52.026701    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6rp2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 31 04:10:52.027237 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:52.026764    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d" podUID="10f0f3ad-038c-4104-ad7c-c0354f33862a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 31 04:10:52.277319 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:52.277230    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-jg5sl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Oct 31 04:10:52.277478 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:52.277317    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl" podUID="86e324aa-c2e3-4524-a3f0-2d2af88b6063" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Oct 31 04:10:53.091225 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:53.091192    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:53.092266 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:53.092241    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:53.093175 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:53.093155    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d"
Oct 31 04:10:53.423349 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:53.423268    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:53.423657 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:53.423633    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:53.424664 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:53.424642    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:10:53.528170 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:53.528121    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk" podUID="2f956311-c81e-4977-bfa9-d1084488abeb" containerName="payload-processing" containerID="cri-o://56fdc4bf5e863905abdaf662dbf3df02e71688f4f1600730622022f029bd6d12" gracePeriod=25
Oct 31 04:10:53.812882 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:53.812846    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"
Oct 31 04:10:53.955015 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:53.954971    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f956311-c81e-4977-bfa9-d1084488abeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f956311-c81e-4977-bfa9-d1084488abeb-plugins-config-volume\") pod \"2f956311-c81e-4977-bfa9-d1084488abeb\" (UID: \"2f956311-c81e-4977-bfa9-d1084488abeb\") "
Oct 31 04:10:53.955196 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:53.955154    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f956311-c81e-4977-bfa9-d1084488abeb-kube-api-access-tjrxk\" (UniqueName: \"kubernetes.io/projected/2f956311-c81e-4977-bfa9-d1084488abeb-kube-api-access-tjrxk\") pod \"2f956311-c81e-4977-bfa9-d1084488abeb\" (UID: \"2f956311-c81e-4977-bfa9-d1084488abeb\") "
Oct 31 04:10:53.955360 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:53.955336    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f956311-c81e-4977-bfa9-d1084488abeb-plugins-config-volume" pod "2f956311-c81e-4977-bfa9-d1084488abeb" (UID: "2f956311-c81e-4977-bfa9-d1084488abeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:53.955653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:53.955636    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f956311-c81e-4977-bfa9-d1084488abeb-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:53.957326 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:53.957296    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f956311-c81e-4977-bfa9-d1084488abeb-kube-api-access-tjrxk" pod "2f956311-c81e-4977-bfa9-d1084488abeb" (UID: "2f956311-c81e-4977-bfa9-d1084488abeb"). InnerVolumeSpecName "kube-api-access-tjrxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:54.058375 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:54.058342    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjrxk\" (UniqueName: \"kubernetes.io/projected/2f956311-c81e-4977-bfa9-d1084488abeb-kube-api-access-tjrxk\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:54.152228 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:54.152193    4900 generic.go:356] "Generic (PLEG): container finished" podID="2f956311-c81e-4977-bfa9-d1084488abeb" containerID="56fdc4bf5e863905abdaf662dbf3df02e71688f4f1600730622022f029bd6d12" exitCode=0
Oct 31 04:10:54.152738 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:54.152237    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"
Oct 31 04:10:54.152738 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:54.152235    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk" event={"ID":"2f956311-c81e-4977-bfa9-d1084488abeb","Type":"ContainerDied","Data":"56fdc4bf5e863905abdaf662dbf3df02e71688f4f1600730622022f029bd6d12"}
Oct 31 04:10:54.152738 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:54.152311    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk" event={"ID":"2f956311-c81e-4977-bfa9-d1084488abeb","Type":"ContainerDied","Data":"dd9c47365d4d043bb06748a7cef9e9ad8e847f954e97c7ff4e09bb9e6e84d6fe"}
Oct 31 04:10:54.152738 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:54.152346    4900 scope.go:122] "RemoveContainer" containerID="56fdc4bf5e863905abdaf662dbf3df02e71688f4f1600730622022f029bd6d12"
Oct 31 04:10:54.162681 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:54.162649    4900 scope.go:122] "RemoveContainer" containerID="56fdc4bf5e863905abdaf662dbf3df02e71688f4f1600730622022f029bd6d12"
Oct 31 04:10:54.162969 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:10:54.162953    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fdc4bf5e863905abdaf662dbf3df02e71688f4f1600730622022f029bd6d12\": container with ID starting with 56fdc4bf5e863905abdaf662dbf3df02e71688f4f1600730622022f029bd6d12 not found: ID does not exist" containerID="56fdc4bf5e863905abdaf662dbf3df02e71688f4f1600730622022f029bd6d12"
Oct 31 04:10:54.163041 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:54.162975    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fdc4bf5e863905abdaf662dbf3df02e71688f4f1600730622022f029bd6d12"} err="failed to get container status \"56fdc4bf5e863905abdaf662dbf3df02e71688f4f1600730622022f029bd6d12\": rpc error: code = NotFound desc = could not find container \"56fdc4bf5e863905abdaf662dbf3df02e71688f4f1600730622022f029bd6d12\": container with ID starting with 56fdc4bf5e863905abdaf662dbf3df02e71688f4f1600730622022f029bd6d12 not found: ID does not exist"
Oct 31 04:10:54.356067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:54.356035    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"]
Oct 31 04:10:54.374015 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:54.373984    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-cfbb5b-6665b44d47-zddlk"]
Oct 31 04:10:54.858499 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:54.858463    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:55.069473 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:55.069442    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:55.155076 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:55.154996    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f956311-c81e-4977-bfa9-d1084488abeb" path="/var/lib/kubelet/pods/2f956311-c81e-4977-bfa9-d1084488abeb/volumes"
Oct 31 04:10:56.518990 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:56.518959    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:56.728467 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:56.728437    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:56.923124 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:56.923044    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"]
Oct 31 04:10:56.923282 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:56.923258    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s" podUID="0187126e-af1f-44bf-9cef-13217cb07672" containerName="istio-proxy" containerID="cri-o://31a2ee59594170621bdd9188a570c6858af3d9e91d04373ab067db7238ed391c" gracePeriod=30
Oct 31 04:10:57.014837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:57.014802    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:58.109328 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:58.109301    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"
Oct 31 04:10:59.126971 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:59.126937    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:10:59.347918 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:59.347883    4900 patch_prober.go:29] interesting pod/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:10:59.348090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:10:59.347939    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s" podUID="0187126e-af1f-44bf-9cef-13217cb07672" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:11:00.045062 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:00.045031    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:00.262072 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:00.262039    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:01.804506 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:01.804472    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:02.197832 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.197803    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:11:02.200712 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.200678    4900 generic.go:356] "Generic (PLEG): container finished" podID="0187126e-af1f-44bf-9cef-13217cb07672" containerID="31a2ee59594170621bdd9188a570c6858af3d9e91d04373ab067db7238ed391c" exitCode=0
Oct 31 04:11:02.200854 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.200726    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"
Oct 31 04:11:02.200854 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.200756    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s" event={"ID":"0187126e-af1f-44bf-9cef-13217cb07672","Type":"ContainerDied","Data":"31a2ee59594170621bdd9188a570c6858af3d9e91d04373ab067db7238ed391c"}
Oct 31 04:11:02.200941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.200865    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s" event={"ID":"0187126e-af1f-44bf-9cef-13217cb07672","Type":"ContainerDied","Data":"bedd9fe14fe772958382e8eded601af761a6bc9fc97d862965ada452bf91d0d3"}
Oct 31 04:11:02.200941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.200890    4900 scope.go:122] "RemoveContainer" containerID="31a2ee59594170621bdd9188a570c6858af3d9e91d04373ab067db7238ed391c"
Oct 31 04:11:02.210550 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.210525    4900 scope.go:122] "RemoveContainer" containerID="69f01b93e95d8198bc82087298eff196179c18dba7f92ebc9d4d6904de6b10aa"
Oct 31 04:11:02.219488 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.219460    4900 scope.go:122] "RemoveContainer" containerID="31a2ee59594170621bdd9188a570c6858af3d9e91d04373ab067db7238ed391c"
Oct 31 04:11:02.219750 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:11:02.219723    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a2ee59594170621bdd9188a570c6858af3d9e91d04373ab067db7238ed391c\": container with ID starting with 31a2ee59594170621bdd9188a570c6858af3d9e91d04373ab067db7238ed391c not found: ID does not exist" containerID="31a2ee59594170621bdd9188a570c6858af3d9e91d04373ab067db7238ed391c"
Oct 31 04:11:02.219808 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.219759    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a2ee59594170621bdd9188a570c6858af3d9e91d04373ab067db7238ed391c"} err="failed to get container status \"31a2ee59594170621bdd9188a570c6858af3d9e91d04373ab067db7238ed391c\": rpc error: code = NotFound desc = could not find container \"31a2ee59594170621bdd9188a570c6858af3d9e91d04373ab067db7238ed391c\": container with ID starting with 31a2ee59594170621bdd9188a570c6858af3d9e91d04373ab067db7238ed391c not found: ID does not exist"
Oct 31 04:11:02.219808 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.219777    4900 scope.go:122] "RemoveContainer" containerID="69f01b93e95d8198bc82087298eff196179c18dba7f92ebc9d4d6904de6b10aa"
Oct 31 04:11:02.220045 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:11:02.220029    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f01b93e95d8198bc82087298eff196179c18dba7f92ebc9d4d6904de6b10aa\": container with ID starting with 69f01b93e95d8198bc82087298eff196179c18dba7f92ebc9d4d6904de6b10aa not found: ID does not exist" containerID="69f01b93e95d8198bc82087298eff196179c18dba7f92ebc9d4d6904de6b10aa"
Oct 31 04:11:02.220088 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.220051    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f01b93e95d8198bc82087298eff196179c18dba7f92ebc9d4d6904de6b10aa"} err="failed to get container status \"69f01b93e95d8198bc82087298eff196179c18dba7f92ebc9d4d6904de6b10aa\": rpc error: code = NotFound desc = could not find container \"69f01b93e95d8198bc82087298eff196179c18dba7f92ebc9d4d6904de6b10aa\": container with ID starting with 69f01b93e95d8198bc82087298eff196179c18dba7f92ebc9d4d6904de6b10aa not found: ID does not exist"
Oct 31 04:11:02.283893 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.283860    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0187126e-af1f-44bf-9cef-13217cb07672-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0187126e-af1f-44bf-9cef-13217cb07672-istio-podinfo\") pod \"0187126e-af1f-44bf-9cef-13217cb07672\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") "
Oct 31 04:11:02.284093 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.283903    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0187126e-af1f-44bf-9cef-13217cb07672-kube-api-access-l6g9j\" (UniqueName: \"kubernetes.io/projected/0187126e-af1f-44bf-9cef-13217cb07672-kube-api-access-l6g9j\") pod \"0187126e-af1f-44bf-9cef-13217cb07672\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") "
Oct 31 04:11:02.284093 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.283996    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0187126e-af1f-44bf-9cef-13217cb07672-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0187126e-af1f-44bf-9cef-13217cb07672-istiod-ca-cert\") pod \"0187126e-af1f-44bf-9cef-13217cb07672\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") "
Oct 31 04:11:02.284093 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.284040    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0187126e-af1f-44bf-9cef-13217cb07672-istio-token\" (UniqueName: \"kubernetes.io/projected/0187126e-af1f-44bf-9cef-13217cb07672-istio-token\") pod \"0187126e-af1f-44bf-9cef-13217cb07672\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") "
Oct 31 04:11:02.284290 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.284146    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-istio-data\") pod \"0187126e-af1f-44bf-9cef-13217cb07672\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") "
Oct 31 04:11:02.284290 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.284190    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-credential-socket\") pod \"0187126e-af1f-44bf-9cef-13217cb07672\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") "
Oct 31 04:11:02.284417 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.284356    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-istio-envoy\") pod \"0187126e-af1f-44bf-9cef-13217cb07672\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") "
Oct 31 04:11:02.284474 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.284432    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-workload-socket\") pod \"0187126e-af1f-44bf-9cef-13217cb07672\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") "
Oct 31 04:11:02.284474 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.284448    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-istio-data" pod "0187126e-af1f-44bf-9cef-13217cb07672" (UID: "0187126e-af1f-44bf-9cef-13217cb07672"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:11:02.284588 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.284485    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0187126e-af1f-44bf-9cef-13217cb07672-istiod-ca-cert" pod "0187126e-af1f-44bf-9cef-13217cb07672" (UID: "0187126e-af1f-44bf-9cef-13217cb07672"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:11:02.284588 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.284498    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-credential-socket" pod "0187126e-af1f-44bf-9cef-13217cb07672" (UID: "0187126e-af1f-44bf-9cef-13217cb07672"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:11:02.284588 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.284504    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-workload-certs\") pod \"0187126e-af1f-44bf-9cef-13217cb07672\" (UID: \"0187126e-af1f-44bf-9cef-13217cb07672\") "
Oct 31 04:11:02.284768 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.284723    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-workload-socket" pod "0187126e-af1f-44bf-9cef-13217cb07672" (UID: "0187126e-af1f-44bf-9cef-13217cb07672"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:11:02.284768 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.284737    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-workload-certs" pod "0187126e-af1f-44bf-9cef-13217cb07672" (UID: "0187126e-af1f-44bf-9cef-13217cb07672"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:11:02.285461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.285434    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:02.285601 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.285465    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:02.285601 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.285481    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0187126e-af1f-44bf-9cef-13217cb07672-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:02.285601 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.285495    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:02.285601 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.285511    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:02.286410 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.286384    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0187126e-af1f-44bf-9cef-13217cb07672-kube-api-access-l6g9j" pod "0187126e-af1f-44bf-9cef-13217cb07672" (UID: "0187126e-af1f-44bf-9cef-13217cb07672"). InnerVolumeSpecName "kube-api-access-l6g9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:11:02.286495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.286386    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0187126e-af1f-44bf-9cef-13217cb07672-istio-podinfo" pod "0187126e-af1f-44bf-9cef-13217cb07672" (UID: "0187126e-af1f-44bf-9cef-13217cb07672"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:11:02.286495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.286409    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0187126e-af1f-44bf-9cef-13217cb07672-istio-token" pod "0187126e-af1f-44bf-9cef-13217cb07672" (UID: "0187126e-af1f-44bf-9cef-13217cb07672"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:11:02.286596 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.286532    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-istio-envoy" pod "0187126e-af1f-44bf-9cef-13217cb07672" (UID: "0187126e-af1f-44bf-9cef-13217cb07672"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:11:02.386956 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.386869    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0187126e-af1f-44bf-9cef-13217cb07672-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:02.386956 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.386900    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0187126e-af1f-44bf-9cef-13217cb07672-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:02.386956 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.386914    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6g9j\" (UniqueName: \"kubernetes.io/projected/0187126e-af1f-44bf-9cef-13217cb07672-kube-api-access-l6g9j\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:02.386956 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.386924    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0187126e-af1f-44bf-9cef-13217cb07672-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:02.622986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.622947    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"]
Oct 31 04:11:02.625397 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:02.625374    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-cfbb5b-openshift-default-584b998857-frx4s"]
Oct 31 04:11:03.153590 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:03.153554    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0187126e-af1f-44bf-9cef-13217cb07672" path="/var/lib/kubelet/pods/0187126e-af1f-44bf-9cef-13217cb07672/volumes"
Oct 31 04:11:03.918727 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:03.918691    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:04.657910 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:04.657876    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:04.885648 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:04.885603    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:06.739617 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.739586    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"]
Oct 31 04:11:06.740204 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.740187    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0187126e-af1f-44bf-9cef-13217cb07672" containerName="istio-proxy"
Oct 31 04:11:06.740258 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.740205    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0187126e-af1f-44bf-9cef-13217cb07672" containerName="istio-proxy"
Oct 31 04:11:06.740258 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.740229    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0187126e-af1f-44bf-9cef-13217cb07672" containerName="istio-proxy"
Oct 31 04:11:06.740258 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.740234    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0187126e-af1f-44bf-9cef-13217cb07672" containerName="istio-proxy"
Oct 31 04:11:06.740258 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.740253    4900 cpu_manager.go:414] "containerMap: removing container" podUID="2f956311-c81e-4977-bfa9-d1084488abeb" containerName="payload-processing"
Oct 31 04:11:06.740258 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.740258    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f956311-c81e-4977-bfa9-d1084488abeb" containerName="payload-processing"
Oct 31 04:11:06.740425 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.740341    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0187126e-af1f-44bf-9cef-13217cb07672" containerName="istio-proxy"
Oct 31 04:11:06.740425 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.740352    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0187126e-af1f-44bf-9cef-13217cb07672" containerName="istio-proxy"
Oct 31 04:11:06.740425 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.740361    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f956311-c81e-4977-bfa9-d1084488abeb" containerName="payload-processing"
Oct 31 04:11:06.743690 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.743673    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:06.795856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.795823    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"]
Oct 31 04:11:06.842032 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.841972    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n7s5\" (UniqueName: \"kubernetes.io/projected/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-kube-api-access-5n7s5\") pod \"maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:06.842206 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.842094    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-maas-api-tls\") pod \"maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:06.842261 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.842248    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-metrics-certs\") pod \"maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:06.842303 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.842293    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:06.842833 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.842816    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f0e007-serving-cert\""
Oct 31 04:11:06.943103 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.943062    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-metrics-certs\") pod \"maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:06.943315 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.943144    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:06.943315 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.943245    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n7s5\" (UniqueName: \"kubernetes.io/projected/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-kube-api-access-5n7s5\") pod \"maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:06.943315 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.943288    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-maas-api-tls\") pod \"maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:06.943493 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:11:06.943460    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-serving-cert: secret "maas-api-e2e-inf-a-f0e007-serving-cert" not found
Oct 31 04:11:06.943559 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:11:06.943547    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-maas-api-tls podName:0cbb8e5d-5188-4305-85dc-a7ee1b3b08df nodeName:}" failed. No retries permitted until 2026-10-31 04:11:07.443527213 +0000 UTC m=+3820.409860537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-maas-api-tls") pod "maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq" (UID: "0cbb8e5d-5188-4305-85dc-a7ee1b3b08df") : secret "maas-api-e2e-inf-a-f0e007-serving-cert" not found
Oct 31 04:11:06.943952 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.943923    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:06.945759 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:06.945732    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-metrics-certs\") pod \"maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:06.985562 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:11:06.985520    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:11:07.007062 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:07.006988    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n7s5\" (UniqueName: \"kubernetes.io/projected/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-kube-api-access-5n7s5\") pod \"maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:07.008998 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:07.008971    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:07.222011 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:07.221978    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:07.434812 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:07.434704    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:07.451377 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:07.451343    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-maas-api-tls\") pod \"maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:07.453861 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:07.453837    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-maas-api-tls\") pod \"maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:07.657092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:07.657057    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:07.831949 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:07.831900    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:08.032151 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:11:08.032112    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbb8e5d_5188_4305_85dc_a7ee1b3b08df.slice/crio-80249ce04c1dc9c11b92be15266d8a8ab96e35ee67a03d446c31eac160d60f80 WatchSource:0}: Error finding container 80249ce04c1dc9c11b92be15266d8a8ab96e35ee67a03d446c31eac160d60f80: Status 404 returned error can't find the container with id 80249ce04c1dc9c11b92be15266d8a8ab96e35ee67a03d446c31eac160d60f80
Oct 31 04:11:08.033812 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.033766    4900 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:11:08.034953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.034906    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"]
Oct 31 04:11:08.035071 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.035045    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:08.243381 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.243346    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq" event={"ID":"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df","Type":"ContainerStarted","Data":"80249ce04c1dc9c11b92be15266d8a8ab96e35ee67a03d446c31eac160d60f80"}
Oct 31 04:11:08.330169 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.330078    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"]
Oct 31 04:11:08.335608 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.335583    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"
Oct 31 04:11:08.385734 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.385706    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"]
Oct 31 04:11:08.388110 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.388087    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d247af\""
Oct 31 04:11:08.407863 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.407833    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a6452c-8457-4c7d-94e3-ac0645d2c5e4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6\" (UID: \"89a6452c-8457-4c7d-94e3-ac0645d2c5e4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"
Oct 31 04:11:08.408033 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.407960    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhb2t\" (UniqueName: \"kubernetes.io/projected/89a6452c-8457-4c7d-94e3-ac0645d2c5e4-kube-api-access-qhb2t\") pod \"payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6\" (UID: \"89a6452c-8457-4c7d-94e3-ac0645d2c5e4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"
Oct 31 04:11:08.432692 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.432657    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d247af-dockercfg-jsr7w\""
Oct 31 04:11:08.508917 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.508886    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a6452c-8457-4c7d-94e3-ac0645d2c5e4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6\" (UID: \"89a6452c-8457-4c7d-94e3-ac0645d2c5e4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"
Oct 31 04:11:08.509095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.508975    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhb2t\" (UniqueName: \"kubernetes.io/projected/89a6452c-8457-4c7d-94e3-ac0645d2c5e4-kube-api-access-qhb2t\") pod \"payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6\" (UID: \"89a6452c-8457-4c7d-94e3-ac0645d2c5e4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"
Oct 31 04:11:08.509607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.509585    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a6452c-8457-4c7d-94e3-ac0645d2c5e4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6\" (UID: \"89a6452c-8457-4c7d-94e3-ac0645d2c5e4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"
Oct 31 04:11:08.566669 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.566641    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhb2t\" (UniqueName: \"kubernetes.io/projected/89a6452c-8457-4c7d-94e3-ac0645d2c5e4-kube-api-access-qhb2t\") pod \"payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6\" (UID: \"89a6452c-8457-4c7d-94e3-ac0645d2c5e4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"
Oct 31 04:11:08.651408 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:08.651332    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"
Oct 31 04:11:09.021724 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:11:09.021695    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a6452c_8457_4c7d_94e3_ac0645d2c5e4.slice/crio-77e096f56c3bbbab024c80cb4db20dca2ba7e33be97b2869d093e53b4109e0c7 WatchSource:0}: Error finding container 77e096f56c3bbbab024c80cb4db20dca2ba7e33be97b2869d093e53b4109e0c7: Status 404 returned error can't find the container with id 77e096f56c3bbbab024c80cb4db20dca2ba7e33be97b2869d093e53b4109e0c7
Oct 31 04:11:09.023586 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:09.023564    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"]
Oct 31 04:11:09.250619 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:09.250518    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6" event={"ID":"89a6452c-8457-4c7d-94e3-ac0645d2c5e4","Type":"ContainerStarted","Data":"a47aa58028850ed227a5657447fc32715f88fc1c59b5422fda017550f2660fd2"}
Oct 31 04:11:09.250619 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:09.250583    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6" event={"ID":"89a6452c-8457-4c7d-94e3-ac0645d2c5e4","Type":"ContainerStarted","Data":"77e096f56c3bbbab024c80cb4db20dca2ba7e33be97b2869d093e53b4109e0c7"}
Oct 31 04:11:09.250619 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:09.250611    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"
Oct 31 04:11:09.252330 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:09.252300    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq" event={"ID":"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df","Type":"ContainerStarted","Data":"45d2d4051f6ed02a15cbf856afaa6d6c5f8e87ecb08aaea16911f8eddcbe196f"}
Oct 31 04:11:09.252472 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:09.252417    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:09.367460 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:09.367402    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6" podStartSLOduration=1.367388153 podStartE2EDuration="1.367388153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:09.365230031 +0000 UTC m=+3822.331563374" watchObservedRunningTime="2026-10-31 04:11:09.367388153 +0000 UTC m=+3822.333721498"
Oct 31 04:11:09.482740 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:09.482675    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq" podStartSLOduration=2.518659806 podStartE2EDuration="3.482651889s" totalImagesPullingTime="963.992083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:09.480023519 +0000 UTC m=+3822.446356864" watchObservedRunningTime="2026-10-31 04:11:09.482651889 +0000 UTC m=+3822.448985251"
Oct 31 04:11:12.048650 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:12.048618    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:14.002267 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:14.002224    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:15.264903 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:15.264868    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:11:18.812459 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:18.812417    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:19.602262 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:19.602219    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:19.811482 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:19.811446    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:20.262173 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:20.262147    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"
Oct 31 04:11:21.241032 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:21.240995    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:21.762265 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:21.762227    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:22.599773 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:22.599740    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:24.103653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:24.103618    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:24.329612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:24.329579    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:25.198471 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:25.198436    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:25.443799 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:25.443752    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:29.511014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:29.510972    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:29.803879 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:29.803798    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:30.019111 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:30.019081    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:33.153243 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:33.153212    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:33.363090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:33.363057    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:34.988073 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:34.988036    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:36.984725 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:11:36.984676    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:11:40.334304 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:40.334254    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:40.559970 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:40.559935    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:40.789023 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:40.788988    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:45.984279 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:45.984242    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:51.379712 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:51.379678    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:56.702936 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:56.702896    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:11:56.903244 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:11:56.903209    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:02.124330 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:02.124280    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:04.261546 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:04.261506    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:06.078280 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:06.078245    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:06.747818 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:06.746729    4900 patch_prober.go:29] interesting pod/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": read tcp 10.133.0.2:35462->10.133.0.164:15021: read: connection reset by peer" start-of-body=
Oct 31 04:12:06.747818 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:06.746822    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm" podUID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": read tcp 10.133.0.2:35462->10.133.0.164:15021: read: connection reset by peer"
Oct 31 04:12:06.987296 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:12:06.987264    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:12:07.092880 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:07.092755    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:07.645370 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:07.645340    4900 generic.go:356] "Generic (PLEG): container finished" podID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerID="d9966e8ef70e83ce0d49a439548259d433ffdde2d3e511c4de1cf6c65be1849f" exitCode=0
Oct 31 04:12:07.645553 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:07.645418    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm" event={"ID":"7af499b8-dbd8-4a13-b009-af4aff66fc7b","Type":"ContainerDied","Data":"d9966e8ef70e83ce0d49a439548259d433ffdde2d3e511c4de1cf6c65be1849f"}
Oct 31 04:12:07.645866 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:07.645851    4900 scope.go:122] "RemoveContainer" containerID="d9966e8ef70e83ce0d49a439548259d433ffdde2d3e511c4de1cf6c65be1849f"
Oct 31 04:12:07.647426 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:07.647399    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:12:07.647508 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:07.647471    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:12:07.647549 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:07.647519    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:12:08.664434 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:08.664395    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm" event={"ID":"7af499b8-dbd8-4a13-b009-af4aff66fc7b","Type":"ContainerStarted","Data":"989718ee4bda586f10574b4fa4cf340d5c274f5f355fc9d19bfef17b674aa175"}
Oct 31 04:12:08.804135 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:08.804107    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:12:08.804325 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:08.804262    4900 patch_prober.go:29] interesting pod/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 04:12:08.804325 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:08.804305    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm" podUID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 04:12:09.804754 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:09.804672    4900 patch_prober.go:29] interesting pod/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 04:12:09.804754 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:09.804732    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm" podUID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 04:12:10.804071 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:10.804038    4900 patch_prober.go:29] interesting pod/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 04:12:10.804251 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:10.804089    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm" podUID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 04:12:11.804221 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:11.804186    4900 patch_prober.go:29] interesting pod/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 04:12:11.804646 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:11.804240    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm" podUID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 04:12:11.987254 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:11.987164    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:12.675686 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:12.675648    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:12.804601 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:12.804569    4900 patch_prober.go:29] interesting pod/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body=
Oct 31 04:12:12.805043 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:12.804632    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm" podUID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused"
Oct 31 04:12:12.881695 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:12.881659    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:13.827963 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:13.827929    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:12:13.828468 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:13.828448    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:12:13.829101 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:13.829083    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:12:16.853088 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:16.853054    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:21.692396 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:21.692362    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:26.338003 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.337967    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"]
Oct 31 04:12:26.342271 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.342244    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:26.396848 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.396815    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"]
Oct 31 04:12:26.443896 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.443868    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d247af-serving-cert\""
Oct 31 04:12:26.444256 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.444233    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p822s\" (UniqueName: \"kubernetes.io/projected/9ce42511-8082-4e74-9822-85d39f2536f3-kube-api-access-p822s\") pod \"maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:26.444299 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.444273    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-metrics-certs\") pod \"maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:26.444370 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.444352    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce42511-8082-4e74-9822-85d39f2536f3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:26.444465 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.444452    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:26.545843 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.545768    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p822s\" (UniqueName: \"kubernetes.io/projected/9ce42511-8082-4e74-9822-85d39f2536f3-kube-api-access-p822s\") pod \"maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:26.546051 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.545854    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-metrics-certs\") pod \"maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:26.546051 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.545894    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce42511-8082-4e74-9822-85d39f2536f3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:26.546051 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.545979    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:26.546212 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:12:26.546145    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-serving-cert: secret "maas-api-e2e-extmod-a-d247af-serving-cert" not found
Oct 31 04:12:26.546269 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:12:26.546218    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-maas-api-tls podName:9ce42511-8082-4e74-9822-85d39f2536f3 nodeName:}" failed. No retries permitted until 2026-10-31 04:12:27.046200786 +0000 UTC m=+3900.012534109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-maas-api-tls") pod "maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn" (UID: "9ce42511-8082-4e74-9822-85d39f2536f3") : secret "maas-api-e2e-extmod-a-d247af-serving-cert" not found
Oct 31 04:12:26.546735 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.546715    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce42511-8082-4e74-9822-85d39f2536f3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:26.548268 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.548251    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-metrics-certs\") pod \"maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:26.604638 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.604566    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p822s\" (UniqueName: \"kubernetes.io/projected/9ce42511-8082-4e74-9822-85d39f2536f3-kube-api-access-p822s\") pod \"maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:26.608964 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.608934    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:26.811599 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:26.811564    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:27.014963 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:27.014932    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:27.052742 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:27.052710    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:27.055259 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:27.055236    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:27.255864 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:27.255828    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:27.623543 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:12:27.623513    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce42511_8082_4e74_9822_85d39f2536f3.slice/crio-1f3203671d05e944f95f3d2e19e3d390fc0ae05c2b1f2a862b6db92a041dba0a WatchSource:0}: Error finding container 1f3203671d05e944f95f3d2e19e3d390fc0ae05c2b1f2a862b6db92a041dba0a: Status 404 returned error can't find the container with id 1f3203671d05e944f95f3d2e19e3d390fc0ae05c2b1f2a862b6db92a041dba0a
Oct 31 04:12:27.625876 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:27.625852    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"]
Oct 31 04:12:27.786782 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:27.786748    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn" event={"ID":"9ce42511-8082-4e74-9822-85d39f2536f3","Type":"ContainerStarted","Data":"1f3203671d05e944f95f3d2e19e3d390fc0ae05c2b1f2a862b6db92a041dba0a"}
Oct 31 04:12:28.794446 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:28.794412    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn" event={"ID":"9ce42511-8082-4e74-9822-85d39f2536f3","Type":"ContainerStarted","Data":"340638882f81b1dcd0aae10efc60c19e8a512cb2ec7d0e547daf33752d1a1563"}
Oct 31 04:12:28.794830 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:28.794541    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:28.865770 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:28.865737    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:12:28.880283 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:28.880222    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:12:28.922726 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:28.922684    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn" podStartSLOduration=1.92487021 podStartE2EDuration="2.922668379s" totalImagesPullingTime="997.798169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:28.920924923 +0000 UTC m=+3901.887258266" watchObservedRunningTime="2026-10-31 04:12:28.922668379 +0000 UTC m=+3901.889001723"
Oct 31 04:12:31.797908 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:31.797875    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:34.806366 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:34.806337    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:12:36.775576 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:36.775541    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:36.983920 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:12:36.983884    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:12:38.920343 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:38.920311    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:39.385941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:39.385907    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:40.162020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:40.161988    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:40.393049 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:40.393012    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:42.358160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:42.358128    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:44.575645 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:44.575611    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:44.813340 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:44.813305    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:47.632251 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:47.632217    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:47.869227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:47.869193    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:48.092894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:48.092861    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:48.466079 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:48.465999    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"]
Oct 31 04:12:48.470092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:48.470074    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"
Oct 31 04:12:48.522469 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:48.522446    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"]
Oct 31 04:12:48.554042 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:48.554015    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj5ch\" (UniqueName: \"kubernetes.io/projected/70848e1b-1960-4cf9-bf4e-fabd84a3e16d-kube-api-access-bj5ch\") pod \"payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r\" (UID: \"70848e1b-1960-4cf9-bf4e-fabd84a3e16d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"
Oct 31 04:12:48.554332 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:48.554243    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70848e1b-1960-4cf9-bf4e-fabd84a3e16d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r\" (UID: \"70848e1b-1960-4cf9-bf4e-fabd84a3e16d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"
Oct 31 04:12:48.569434 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:48.569406    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-243a29\""
Oct 31 04:12:48.569550 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:48.569489    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-243a29-dockercfg-9qlp8\""
Oct 31 04:12:48.655174 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:48.655144    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70848e1b-1960-4cf9-bf4e-fabd84a3e16d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r\" (UID: \"70848e1b-1960-4cf9-bf4e-fabd84a3e16d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"
Oct 31 04:12:48.655514 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:48.655315    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5ch\" (UniqueName: \"kubernetes.io/projected/70848e1b-1960-4cf9-bf4e-fabd84a3e16d-kube-api-access-bj5ch\") pod \"payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r\" (UID: \"70848e1b-1960-4cf9-bf4e-fabd84a3e16d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"
Oct 31 04:12:48.655883 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:48.655864    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70848e1b-1960-4cf9-bf4e-fabd84a3e16d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r\" (UID: \"70848e1b-1960-4cf9-bf4e-fabd84a3e16d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"
Oct 31 04:12:48.712131 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:48.712098    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5ch\" (UniqueName: \"kubernetes.io/projected/70848e1b-1960-4cf9-bf4e-fabd84a3e16d-kube-api-access-bj5ch\") pod \"payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r\" (UID: \"70848e1b-1960-4cf9-bf4e-fabd84a3e16d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"
Oct 31 04:12:48.783021 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:48.782991    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"
Oct 31 04:12:49.152101 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:12:49.152070    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70848e1b_1960_4cf9_bf4e_fabd84a3e16d.slice/crio-c680826195cd1d289de63527b91a7c39868f503b64f3858c36dcc67ba32daaab WatchSource:0}: Error finding container c680826195cd1d289de63527b91a7c39868f503b64f3858c36dcc67ba32daaab: Status 404 returned error can't find the container with id c680826195cd1d289de63527b91a7c39868f503b64f3858c36dcc67ba32daaab
Oct 31 04:12:49.154478 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:49.154456    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"]
Oct 31 04:12:49.918777 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:49.918740    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r" event={"ID":"70848e1b-1960-4cf9-bf4e-fabd84a3e16d","Type":"ContainerStarted","Data":"1fff83e89c4b02bf79f16ca1491cf37d13e07c0c7e7da44c3ec828f5c7203913"}
Oct 31 04:12:49.919211 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:49.918929    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"
Oct 31 04:12:49.919211 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:49.918960    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r" event={"ID":"70848e1b-1960-4cf9-bf4e-fabd84a3e16d","Type":"ContainerStarted","Data":"c680826195cd1d289de63527b91a7c39868f503b64f3858c36dcc67ba32daaab"}
Oct 31 04:12:50.033696 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:50.033641    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r" podStartSLOduration=2.033627268 podStartE2EDuration="2.033627268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:50.031834802 +0000 UTC m=+3922.998168148" watchObservedRunningTime="2026-10-31 04:12:50.033627268 +0000 UTC m=+3922.999960636"
Oct 31 04:12:52.772144 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:52.772109    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:52.986889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:52.986852    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:12:57.797116 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:12:57.797076    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:00.928719 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:00.928680    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"
Oct 31 04:13:02.797107 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:02.797063    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:04.625850 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:04.625815    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:06.985997 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:13:06.985956    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:13:07.662984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:07.662950    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:12.676424 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:12.676392    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:17.492244 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:17.492197    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:17.961432 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:17.961346    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:18.194276 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.194230    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:18.405559 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.405524    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"]
Oct 31 04:13:18.409889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.409871    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.469952 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.469917    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6f30f2-openshift-default-dockercfg-wgfmh\""
Oct 31 04:13:18.539325 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.539294    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"]
Oct 31 04:13:18.547663 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.547629    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-envoy\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.547856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.547680    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-credential-socket\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.547856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.547711    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istiod-ca-cert\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.547856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.547757    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-podinfo\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.548053 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.547876    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-workload-certs\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.548053 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.547996    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-workload-socket\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.548162 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.548065    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-data\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.548162 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.548134    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-token\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.548238 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.548206    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zp5w\" (UniqueName: \"kubernetes.io/projected/e6b94bdc-4427-400c-ad3f-5032aa82a83f-kube-api-access-6zp5w\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.582025 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.581971    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:18.649600 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.649567    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-workload-certs\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.649600 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.649605    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-workload-socket\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.649868 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.649635    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-data\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.649868 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.649655    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-token\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.649868 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.649694    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zp5w\" (UniqueName: \"kubernetes.io/projected/e6b94bdc-4427-400c-ad3f-5032aa82a83f-kube-api-access-6zp5w\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.649868 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.649832    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-envoy\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.650095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.649874    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-credential-socket\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.650095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.649906    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istiod-ca-cert\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.650095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.650031    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-podinfo\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.650095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.650053    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-workload-certs\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.650323 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.650187    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-workload-socket\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.650385 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.650326    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-credential-socket\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.650385 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.650354    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-data\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.650664 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.650646    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istiod-ca-cert\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.652142 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.652118    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-envoy\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.652440 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.652421    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-podinfo\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.707164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.707089    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-token\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.707324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.707192    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zp5w\" (UniqueName: \"kubernetes.io/projected/e6b94bdc-4427-400c-ad3f-5032aa82a83f-kube-api-access-6zp5w\") pod \"e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") " pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.724138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.724104    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:18.813138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:18.813107    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:19.152247 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:13:19.152212    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b94bdc_4427_400c_ad3f_5032aa82a83f.slice/crio-8860a929de4e462255c830f2caacd701bb13c090cdbf2cd2722b02edfa9216e8 WatchSource:0}: Error finding container 8860a929de4e462255c830f2caacd701bb13c090cdbf2cd2722b02edfa9216e8: Status 404 returned error can't find the container with id 8860a929de4e462255c830f2caacd701bb13c090cdbf2cd2722b02edfa9216e8
Oct 31 04:13:19.155423 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:19.155389    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:13:19.155516 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:19.155461    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:13:19.155516 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:19.155493    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:13:19.155714 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:19.155694    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"]
Oct 31 04:13:20.105703 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:20.105659    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px" event={"ID":"e6b94bdc-4427-400c-ad3f-5032aa82a83f","Type":"ContainerStarted","Data":"e8419a3d511e712c305033d9a5f013a54b9a48c9d956ede40006ff61fcaf2fce"}
Oct 31 04:13:20.106233 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:20.105719    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px" event={"ID":"e6b94bdc-4427-400c-ad3f-5032aa82a83f","Type":"ContainerStarted","Data":"8860a929de4e462255c830f2caacd701bb13c090cdbf2cd2722b02edfa9216e8"}
Oct 31 04:13:20.324990 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:20.324939    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px" podStartSLOduration=2.3249237640000002 podStartE2EDuration="2.324923764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:20.274272943 +0000 UTC m=+3953.240606287" watchObservedRunningTime="2026-10-31 04:13:20.324923764 +0000 UTC m=+3953.291257107"
Oct 31 04:13:20.725274 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:20.725240    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:20.731190 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:20.731164    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:21.112371 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:21.112339    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:21.113402 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:21.113383    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:13:23.241068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:23.241033    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:23.476776 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:23.476747    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:23.712030 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:23.711946    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:26.522139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:26.522101    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:26.741241 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:26.741211    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:28.808853 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:28.808817    4900 scope.go:122] "RemoveContainer" containerID="084d2104fd5864660f49ae65a7826e0a582862aa329c68ed3940e0a51640c291"
Oct 31 04:13:29.061248 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:29.061126    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:33.784073 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:33.784032    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:34.328067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.328033    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:34.600075 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.599975    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:34.840577 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.840547    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"]
Oct 31 04:13:34.844576 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.844559    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:34.897684 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.897653    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e679fe-openshift-default-dockercfg-lgw7n\""
Oct 31 04:13:34.962066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.962017    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"]
Oct 31 04:13:34.980529 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.980489    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e37b79-c673-4bb9-a740-eda25789449a-istio-token\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:34.980880 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.980852    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e37b79-c673-4bb9-a740-eda25789449a-istio-podinfo\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:34.981054 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.981036    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-workload-certs\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:34.981232 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.981213    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-istio-data\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:34.981381 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.981364    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-workload-socket\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:34.981528 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.981510    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-credential-socket\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:34.981658 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.981642    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzjh6\" (UniqueName: \"kubernetes.io/projected/c0e37b79-c673-4bb9-a740-eda25789449a-kube-api-access-gzjh6\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:34.981869 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.981848    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e37b79-c673-4bb9-a740-eda25789449a-istiod-ca-cert\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:34.982135 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:34.982117    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-istio-envoy\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.012034 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.012009    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:35.084102 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.084064    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-credential-socket\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.084102 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.084112    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzjh6\" (UniqueName: \"kubernetes.io/projected/c0e37b79-c673-4bb9-a740-eda25789449a-kube-api-access-gzjh6\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.084358 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.084201    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e37b79-c673-4bb9-a740-eda25789449a-istiod-ca-cert\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.084421 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.084376    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-istio-envoy\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.084488 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.084422    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e37b79-c673-4bb9-a740-eda25789449a-istio-token\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.084488 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.084478    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e37b79-c673-4bb9-a740-eda25789449a-istio-podinfo\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.084593 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.084515    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-workload-certs\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.084593 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.084561    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-credential-socket\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.084593 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.084577    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-istio-data\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.084772 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.084645    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-workload-socket\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.084959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.084935    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-istio-data\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.085042 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.085023    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-workload-socket\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.085098 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.085044    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e37b79-c673-4bb9-a740-eda25789449a-istiod-ca-cert\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.086075 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.086041    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-workload-certs\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.086998 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.086980    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-istio-envoy\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.087309 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.087290    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e37b79-c673-4bb9-a740-eda25789449a-istio-podinfo\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.141460 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.141390    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e37b79-c673-4bb9-a740-eda25789449a-istio-token\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.141627 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.141581    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzjh6\" (UniqueName: \"kubernetes.io/projected/c0e37b79-c673-4bb9-a740-eda25789449a-kube-api-access-gzjh6\") pod \"e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") " pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.158285 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.158256    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:35.266372 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.266342    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:35.578661 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:13:35.578629    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e37b79_c673_4bb9_a740_eda25789449a.slice/crio-55637c7d8731b8e9c810831ffb0f5308ffd14d3788b3a370604b2e79eece7543 WatchSource:0}: Error finding container 55637c7d8731b8e9c810831ffb0f5308ffd14d3788b3a370604b2e79eece7543: Status 404 returned error can't find the container with id 55637c7d8731b8e9c810831ffb0f5308ffd14d3788b3a370604b2e79eece7543
Oct 31 04:13:35.581415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.581393    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"]
Oct 31 04:13:35.581493 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.581437    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:13:35.581537 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.581499    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:13:35.581537 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:35.581526    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:13:36.219198 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:36.219158    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq" event={"ID":"c0e37b79-c673-4bb9-a740-eda25789449a","Type":"ContainerStarted","Data":"569f4ec56e74a1e590dc0b2b2981dccbaf482b9ecb0d49aff3ea33e330735a10"}
Oct 31 04:13:36.219620 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:36.219237    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq" event={"ID":"c0e37b79-c673-4bb9-a740-eda25789449a","Type":"ContainerStarted","Data":"55637c7d8731b8e9c810831ffb0f5308ffd14d3788b3a370604b2e79eece7543"}
Oct 31 04:13:36.337449 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:36.337394    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq" podStartSLOduration=2.337380151 podStartE2EDuration="2.337380151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:13:36.334930835 +0000 UTC m=+3969.301264178" watchObservedRunningTime="2026-10-31 04:13:36.337380151 +0000 UTC m=+3969.303713494"
Oct 31 04:13:36.805327 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:36.805291    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:36.985845 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:13:36.985809    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:13:37.020611 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:37.020567    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:37.159385 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:37.159292    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:37.165210 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:37.165187    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:37.225664 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:37.225632    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:37.226628 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:37.226607    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:13:39.463113 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:39.463026    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:39.755744 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:39.755712    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:40.116289 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:40.116203    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:43.681495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:43.681450    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:43.903273 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:43.903229    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:45.503298 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:45.503230    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:50.407799 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:50.407746    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:51.617848 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:51.617814    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:13:55.280136 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:13:55.280101    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:00.280129 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:00.280093    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:02.049010 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:02.048971    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"]
Oct 31 04:14:02.097078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:02.097040    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0b136c3-ds92v"]
Oct 31 04:14:03.154121 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:03.154089    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c410152f-0742-4ff7-87fa-71688aa7df32" path="/var/lib/kubelet/pods/c410152f-0742-4ff7-87fa-71688aa7df32/volumes"
Oct 31 04:14:04.102708 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:04.102662    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:04.967463 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:04.967427    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:05.412562 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:05.412523    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:06.985489 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:14:06.985450    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:14:09.411831 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:09.411781    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"]
Oct 31 04:14:09.869006 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:09.868968    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:10.100458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:10.100431    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:10.393151 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:10.393110    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:14.412264 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:14.412223    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6" podUID="89a6452c-8457-4c7d-94e3-ac0645d2c5e4" containerName="payload-processing" containerID="cri-o://a47aa58028850ed227a5657447fc32715f88fc1c59b5422fda017550f2660fd2" gracePeriod=25
Oct 31 04:14:14.683034 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:14.683003    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"
Oct 31 04:14:14.718076 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:14.718049    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a6452c-8457-4c7d-94e3-ac0645d2c5e4-kube-api-access-qhb2t\" (UniqueName: \"kubernetes.io/projected/89a6452c-8457-4c7d-94e3-ac0645d2c5e4-kube-api-access-qhb2t\") pod \"89a6452c-8457-4c7d-94e3-ac0645d2c5e4\" (UID: \"89a6452c-8457-4c7d-94e3-ac0645d2c5e4\") "
Oct 31 04:14:14.718211 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:14.718133    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a6452c-8457-4c7d-94e3-ac0645d2c5e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a6452c-8457-4c7d-94e3-ac0645d2c5e4-plugins-config-volume\") pod \"89a6452c-8457-4c7d-94e3-ac0645d2c5e4\" (UID: \"89a6452c-8457-4c7d-94e3-ac0645d2c5e4\") "
Oct 31 04:14:14.718509 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:14.718487    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a6452c-8457-4c7d-94e3-ac0645d2c5e4-plugins-config-volume" pod "89a6452c-8457-4c7d-94e3-ac0645d2c5e4" (UID: "89a6452c-8457-4c7d-94e3-ac0645d2c5e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:14:14.720259 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:14.720236    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a6452c-8457-4c7d-94e3-ac0645d2c5e4-kube-api-access-qhb2t" pod "89a6452c-8457-4c7d-94e3-ac0645d2c5e4" (UID: "89a6452c-8457-4c7d-94e3-ac0645d2c5e4"). InnerVolumeSpecName "kube-api-access-qhb2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:14:14.819419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:14.819382    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a6452c-8457-4c7d-94e3-ac0645d2c5e4-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:14.819419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:14.819413    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhb2t\" (UniqueName: \"kubernetes.io/projected/89a6452c-8457-4c7d-94e3-ac0645d2c5e4-kube-api-access-qhb2t\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:15.306185 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:15.306151    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:15.489531 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:15.489494    4900 generic.go:356] "Generic (PLEG): container finished" podID="89a6452c-8457-4c7d-94e3-ac0645d2c5e4" containerID="a47aa58028850ed227a5657447fc32715f88fc1c59b5422fda017550f2660fd2" exitCode=0
Oct 31 04:14:15.490005 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:15.489539    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"
Oct 31 04:14:15.490005 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:15.489583    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6" event={"ID":"89a6452c-8457-4c7d-94e3-ac0645d2c5e4","Type":"ContainerDied","Data":"a47aa58028850ed227a5657447fc32715f88fc1c59b5422fda017550f2660fd2"}
Oct 31 04:14:15.490005 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:15.489632    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6" event={"ID":"89a6452c-8457-4c7d-94e3-ac0645d2c5e4","Type":"ContainerDied","Data":"77e096f56c3bbbab024c80cb4db20dca2ba7e33be97b2869d093e53b4109e0c7"}
Oct 31 04:14:15.490005 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:15.489656    4900 scope.go:122] "RemoveContainer" containerID="a47aa58028850ed227a5657447fc32715f88fc1c59b5422fda017550f2660fd2"
Oct 31 04:14:15.499513 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:15.499495    4900 scope.go:122] "RemoveContainer" containerID="a47aa58028850ed227a5657447fc32715f88fc1c59b5422fda017550f2660fd2"
Oct 31 04:14:15.499771 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:14:15.499750    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a47aa58028850ed227a5657447fc32715f88fc1c59b5422fda017550f2660fd2\": container with ID starting with a47aa58028850ed227a5657447fc32715f88fc1c59b5422fda017550f2660fd2 not found: ID does not exist" containerID="a47aa58028850ed227a5657447fc32715f88fc1c59b5422fda017550f2660fd2"
Oct 31 04:14:15.499920 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:15.499781    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47aa58028850ed227a5657447fc32715f88fc1c59b5422fda017550f2660fd2"} err="failed to get container status \"a47aa58028850ed227a5657447fc32715f88fc1c59b5422fda017550f2660fd2\": rpc error: code = NotFound desc = could not find container \"a47aa58028850ed227a5657447fc32715f88fc1c59b5422fda017550f2660fd2\": container with ID starting with a47aa58028850ed227a5657447fc32715f88fc1c59b5422fda017550f2660fd2 not found: ID does not exist"
Oct 31 04:14:15.516993 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:15.516974    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:15.653305 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:15.653272    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"]
Oct 31 04:14:15.655976 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:15.655957    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d247af-57fc994bb7-p5nv6"]
Oct 31 04:14:15.734158 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:15.734123    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:17.152975 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:17.152941    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a6452c-8457-4c7d-94e3-ac0645d2c5e4" path="/var/lib/kubelet/pods/89a6452c-8457-4c7d-94e3-ac0645d2c5e4/volumes"
Oct 31 04:14:20.498187 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:20.498146    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:25.333331 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:25.333292    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:29.430072 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:29.430027    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:30.281941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:30.281906    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:35.132856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:35.132822    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:35.359625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:35.358818    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:35.585874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:35.585831    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:36.986866 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:14:36.986824    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:14:40.219678 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:40.219636    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:40.775403 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:40.775365    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:45.178456 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:45.178426    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:50.088028 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:50.087987    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:54.841157 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:54.841120    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:55.053272 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:55.053241    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:14:59.897148 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:14:59.897101    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:00.172677 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.172589    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"]
Oct 31 04:15:00.173271 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.173255    4900 cpu_manager.go:414] "containerMap: removing container" podUID="89a6452c-8457-4c7d-94e3-ac0645d2c5e4" containerName="payload-processing"
Oct 31 04:15:00.173322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.173273    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a6452c-8457-4c7d-94e3-ac0645d2c5e4" containerName="payload-processing"
Oct 31 04:15:00.173389 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.173380    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a6452c-8457-4c7d-94e3-ac0645d2c5e4" containerName="payload-processing"
Oct 31 04:15:00.176957 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.176941    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"]
Oct 31 04:15:00.177076 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.177058    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"
Oct 31 04:15:00.181594 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.181572    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"
Oct 31 04:15:00.198010 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.197985    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"]
Oct 31 04:15:00.202131 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.202104    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"
Oct 31 04:15:00.214681 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.214655    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ndr5\" (UniqueName: \"kubernetes.io/projected/64a9e5b0-9d34-468d-a70a-304974436a9b-kube-api-access-5ndr5\") pod \"maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw\" (UID: \"64a9e5b0-9d34-468d-a70a-304974436a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"
Oct 31 04:15:00.214853 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.214702    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2146db6-ed0b-4776-930f-e1f915526586-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r\" (UID: \"a2146db6-ed0b-4776-930f-e1f915526586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"
Oct 31 04:15:00.214853 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.214725    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a9e5b0-9d34-468d-a70a-304974436a9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw\" (UID: \"64a9e5b0-9d34-468d-a70a-304974436a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"
Oct 31 04:15:00.215076 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.215056    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4bbz\" (UniqueName: \"kubernetes.io/projected/e4447ca2-888d-4950-b75d-b6149d71791a-kube-api-access-b4bbz\") pod \"maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm\" (UID: \"e4447ca2-888d-4950-b75d-b6149d71791a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"
Oct 31 04:15:00.215138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.215109    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4447ca2-888d-4950-b75d-b6149d71791a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm\" (UID: \"e4447ca2-888d-4950-b75d-b6149d71791a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"
Oct 31 04:15:00.215191 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.215158    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-246b9\" (UniqueName: \"kubernetes.io/projected/a2146db6-ed0b-4776-930f-e1f915526586-kube-api-access-246b9\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r\" (UID: \"a2146db6-ed0b-4776-930f-e1f915526586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"
Oct 31 04:15:00.222924 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.222897    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"]
Oct 31 04:15:00.227915 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.227894    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"]
Oct 31 04:15:00.228042 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.228024    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"
Oct 31 04:15:00.232611 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.232594    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"]
Oct 31 04:15:00.232739 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.232715    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"
Oct 31 04:15:00.237140 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.237121    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"
Oct 31 04:15:00.243692 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.243674    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:15:00.272914 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.272886    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"]
Oct 31 04:15:00.276894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.276877    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"]
Oct 31 04:15:00.276998 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.276962    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"
Oct 31 04:15:00.278082 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.278058    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"]
Oct 31 04:15:00.309771 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.309737    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"]
Oct 31 04:15:00.312495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.312464    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"]
Oct 31 04:15:00.315444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.315417    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"]
Oct 31 04:15:00.316047 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.316021    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-246b9\" (UniqueName: \"kubernetes.io/projected/a2146db6-ed0b-4776-930f-e1f915526586-kube-api-access-246b9\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r\" (UID: \"a2146db6-ed0b-4776-930f-e1f915526586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"
Oct 31 04:15:00.316142 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.316073    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdplk\" (UniqueName: \"kubernetes.io/projected/713a4870-e144-4960-bbff-0c4e623ba57c-kube-api-access-pdplk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf\" (UID: \"713a4870-e144-4960-bbff-0c4e623ba57c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"
Oct 31 04:15:00.316142 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.316107    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwr5\" (UniqueName: \"kubernetes.io/projected/8797d751-7c89-4eb8-aaf9-abf0c81efcca-kube-api-access-hzwr5\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb\" (UID: \"8797d751-7c89-4eb8-aaf9-abf0c81efcca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"
Oct 31 04:15:00.316142 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.316124    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b45f245-d9e1-4476-bdb9-4a840c078444-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf\" (UID: \"8b45f245-d9e1-4476-bdb9-4a840c078444\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"
Oct 31 04:15:00.316291 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.316271    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8797d751-7c89-4eb8-aaf9-abf0c81efcca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb\" (UID: \"8797d751-7c89-4eb8-aaf9-abf0c81efcca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"
Oct 31 04:15:00.316334 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.316324    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713a4870-e144-4960-bbff-0c4e623ba57c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf\" (UID: \"713a4870-e144-4960-bbff-0c4e623ba57c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"
Oct 31 04:15:00.316405 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.316392    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ndr5\" (UniqueName: \"kubernetes.io/projected/64a9e5b0-9d34-468d-a70a-304974436a9b-kube-api-access-5ndr5\") pod \"maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw\" (UID: \"64a9e5b0-9d34-468d-a70a-304974436a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"
Oct 31 04:15:00.316450 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.316417    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfcb4c4-007c-4acc-a20c-092b12cfc44b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw\" (UID: \"ecfcb4c4-007c-4acc-a20c-092b12cfc44b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"
Oct 31 04:15:00.316450 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.316435    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2146db6-ed0b-4776-930f-e1f915526586-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r\" (UID: \"a2146db6-ed0b-4776-930f-e1f915526586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"
Oct 31 04:15:00.316577 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.316561    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a9e5b0-9d34-468d-a70a-304974436a9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw\" (UID: \"64a9e5b0-9d34-468d-a70a-304974436a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"
Oct 31 04:15:00.316638 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.316600    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbczp\" (UniqueName: \"kubernetes.io/projected/ecfcb4c4-007c-4acc-a20c-092b12cfc44b-kube-api-access-nbczp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw\" (UID: \"ecfcb4c4-007c-4acc-a20c-092b12cfc44b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"
Oct 31 04:15:00.316712 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.316647    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4bbz\" (UniqueName: \"kubernetes.io/projected/e4447ca2-888d-4950-b75d-b6149d71791a-kube-api-access-b4bbz\") pod \"maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm\" (UID: \"e4447ca2-888d-4950-b75d-b6149d71791a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"
Oct 31 04:15:00.316712 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.316686    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psd5n\" (UniqueName: \"kubernetes.io/projected/8b45f245-d9e1-4476-bdb9-4a840c078444-kube-api-access-psd5n\") pod \"maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf\" (UID: \"8b45f245-d9e1-4476-bdb9-4a840c078444\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"
Oct 31 04:15:00.316814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.316743    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4447ca2-888d-4950-b75d-b6149d71791a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm\" (UID: \"e4447ca2-888d-4950-b75d-b6149d71791a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"
Oct 31 04:15:00.317187 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.317168    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2146db6-ed0b-4776-930f-e1f915526586-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r\" (UID: \"a2146db6-ed0b-4776-930f-e1f915526586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"
Oct 31 04:15:00.317418 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.317397    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4447ca2-888d-4950-b75d-b6149d71791a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm\" (UID: \"e4447ca2-888d-4950-b75d-b6149d71791a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"
Oct 31 04:15:00.317485 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.317400    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a9e5b0-9d34-468d-a70a-304974436a9b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw\" (UID: \"64a9e5b0-9d34-468d-a70a-304974436a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"
Oct 31 04:15:00.366902 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.366867    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"]
Oct 31 04:15:00.371599 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.371565    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"]
Oct 31 04:15:00.379529 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.379500    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ndr5\" (UniqueName: \"kubernetes.io/projected/64a9e5b0-9d34-468d-a70a-304974436a9b-kube-api-access-5ndr5\") pod \"maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw\" (UID: \"64a9e5b0-9d34-468d-a70a-304974436a9b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"
Oct 31 04:15:00.389837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.389363    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-246b9\" (UniqueName: \"kubernetes.io/projected/a2146db6-ed0b-4776-930f-e1f915526586-kube-api-access-246b9\") pod \"maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r\" (UID: \"a2146db6-ed0b-4776-930f-e1f915526586\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"
Oct 31 04:15:00.394332 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.394299    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4bbz\" (UniqueName: \"kubernetes.io/projected/e4447ca2-888d-4950-b75d-b6149d71791a-kube-api-access-b4bbz\") pod \"maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm\" (UID: \"e4447ca2-888d-4950-b75d-b6149d71791a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"
Oct 31 04:15:00.418105 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.418068    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8797d751-7c89-4eb8-aaf9-abf0c81efcca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb\" (UID: \"8797d751-7c89-4eb8-aaf9-abf0c81efcca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"
Oct 31 04:15:00.418300 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.418159    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713a4870-e144-4960-bbff-0c4e623ba57c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf\" (UID: \"713a4870-e144-4960-bbff-0c4e623ba57c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"
Oct 31 04:15:00.418300 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.418216    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfcb4c4-007c-4acc-a20c-092b12cfc44b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw\" (UID: \"ecfcb4c4-007c-4acc-a20c-092b12cfc44b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"
Oct 31 04:15:00.418300 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.418268    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbczp\" (UniqueName: \"kubernetes.io/projected/ecfcb4c4-007c-4acc-a20c-092b12cfc44b-kube-api-access-nbczp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw\" (UID: \"ecfcb4c4-007c-4acc-a20c-092b12cfc44b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"
Oct 31 04:15:00.418521 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.418330    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psd5n\" (UniqueName: \"kubernetes.io/projected/8b45f245-d9e1-4476-bdb9-4a840c078444-kube-api-access-psd5n\") pod \"maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf\" (UID: \"8b45f245-d9e1-4476-bdb9-4a840c078444\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"
Oct 31 04:15:00.418521 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.418471    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdplk\" (UniqueName: \"kubernetes.io/projected/713a4870-e144-4960-bbff-0c4e623ba57c-kube-api-access-pdplk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf\" (UID: \"713a4870-e144-4960-bbff-0c4e623ba57c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"
Oct 31 04:15:00.418628 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.418516    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwr5\" (UniqueName: \"kubernetes.io/projected/8797d751-7c89-4eb8-aaf9-abf0c81efcca-kube-api-access-hzwr5\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb\" (UID: \"8797d751-7c89-4eb8-aaf9-abf0c81efcca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"
Oct 31 04:15:00.418628 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.418543    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b45f245-d9e1-4476-bdb9-4a840c078444-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf\" (UID: \"8b45f245-d9e1-4476-bdb9-4a840c078444\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"
Oct 31 04:15:00.419146 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.419124    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8797d751-7c89-4eb8-aaf9-abf0c81efcca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb\" (UID: \"8797d751-7c89-4eb8-aaf9-abf0c81efcca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"
Oct 31 04:15:00.419272 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.419219    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713a4870-e144-4960-bbff-0c4e623ba57c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf\" (UID: \"713a4870-e144-4960-bbff-0c4e623ba57c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"
Oct 31 04:15:00.419272 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.419235    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b45f245-d9e1-4476-bdb9-4a840c078444-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf\" (UID: \"8b45f245-d9e1-4476-bdb9-4a840c078444\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"
Oct 31 04:15:00.419919 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.419897    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfcb4c4-007c-4acc-a20c-092b12cfc44b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw\" (UID: \"ecfcb4c4-007c-4acc-a20c-092b12cfc44b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"
Oct 31 04:15:00.478110 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.477378    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbczp\" (UniqueName: \"kubernetes.io/projected/ecfcb4c4-007c-4acc-a20c-092b12cfc44b-kube-api-access-nbczp\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw\" (UID: \"ecfcb4c4-007c-4acc-a20c-092b12cfc44b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"
Oct 31 04:15:00.478110 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.477493    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdplk\" (UniqueName: \"kubernetes.io/projected/713a4870-e144-4960-bbff-0c4e623ba57c-kube-api-access-pdplk\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf\" (UID: \"713a4870-e144-4960-bbff-0c4e623ba57c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"
Oct 31 04:15:00.478110 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.477535    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psd5n\" (UniqueName: \"kubernetes.io/projected/8b45f245-d9e1-4476-bdb9-4a840c078444-kube-api-access-psd5n\") pod \"maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf\" (UID: \"8b45f245-d9e1-4476-bdb9-4a840c078444\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"
Oct 31 04:15:00.478110 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.477569    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwr5\" (UniqueName: \"kubernetes.io/projected/8797d751-7c89-4eb8-aaf9-abf0c81efcca-kube-api-access-hzwr5\") pod \"maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb\" (UID: \"8797d751-7c89-4eb8-aaf9-abf0c81efcca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"
Oct 31 04:15:00.492378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.492348    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"
Oct 31 04:15:00.500889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.500864    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"
Oct 31 04:15:00.515077 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.515052    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"
Oct 31 04:15:00.544242 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.544039    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"
Oct 31 04:15:00.559686 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.559528    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"
Oct 31 04:15:00.560660 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.560203    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"
Oct 31 04:15:00.598992 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.598959    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"
Oct 31 04:15:00.868606 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:15:00.868575    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4447ca2_888d_4950_b75d_b6149d71791a.slice/crio-66e7ecf02ab2f3133bf24ed53fc9cb27d40271e5cb7eb08caa9ce5693502ed42 WatchSource:0}: Error finding container 66e7ecf02ab2f3133bf24ed53fc9cb27d40271e5cb7eb08caa9ce5693502ed42: Status 404 returned error can't find the container with id 66e7ecf02ab2f3133bf24ed53fc9cb27d40271e5cb7eb08caa9ce5693502ed42
Oct 31 04:15:00.872186 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.872161    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"]
Oct 31 04:15:00.913513 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:15:00.913481    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2146db6_ed0b_4776_930f_e1f915526586.slice/crio-94c0135df4a746d2e7daf5a086458c8283d3f5695edd3f314c70fefb6688d847 WatchSource:0}: Error finding container 94c0135df4a746d2e7daf5a086458c8283d3f5695edd3f314c70fefb6688d847: Status 404 returned error can't find the container with id 94c0135df4a746d2e7daf5a086458c8283d3f5695edd3f314c70fefb6688d847
Oct 31 04:15:00.916958 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.916930    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"]
Oct 31 04:15:00.976243 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:15:00.976213    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfcb4c4_007c_4acc_a20c_092b12cfc44b.slice/crio-25465c427dd2a77642169114bdd26e6a02997e86dd2547748c5459319e6c49af WatchSource:0}: Error finding container 25465c427dd2a77642169114bdd26e6a02997e86dd2547748c5459319e6c49af: Status 404 returned error can't find the container with id 25465c427dd2a77642169114bdd26e6a02997e86dd2547748c5459319e6c49af
Oct 31 04:15:00.978549 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:00.978525    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"]
Oct 31 04:15:01.033005 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:15:01.032977    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8797d751_7c89_4eb8_aaf9_abf0c81efcca.slice/crio-40ccf3abfe910cabbbd0c2aab68a278c7dc6f60bbec0ef9c10fcc9eeab029564 WatchSource:0}: Error finding container 40ccf3abfe910cabbbd0c2aab68a278c7dc6f60bbec0ef9c10fcc9eeab029564: Status 404 returned error can't find the container with id 40ccf3abfe910cabbbd0c2aab68a278c7dc6f60bbec0ef9c10fcc9eeab029564
Oct 31 04:15:01.037964 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.035683    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"]
Oct 31 04:15:01.057253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.057229    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"]
Oct 31 04:15:01.085066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.085040    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"]
Oct 31 04:15:01.092063 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:15:01.092026    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a9e5b0_9d34_468d_a70a_304974436a9b.slice/crio-26a469e506bfc0145dee2d6eb033c65afde8dbf80b26651ac0884fbe852c6863 WatchSource:0}: Error finding container 26a469e506bfc0145dee2d6eb033c65afde8dbf80b26651ac0884fbe852c6863: Status 404 returned error can't find the container with id 26a469e506bfc0145dee2d6eb033c65afde8dbf80b26651ac0884fbe852c6863
Oct 31 04:15:01.155412 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.155388    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"]
Oct 31 04:15:01.249261 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:15:01.249193    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713a4870_e144_4960_bbff_0c4e623ba57c.slice/crio-f5d7cb0236739c8c9a65c274dd95b566e5575c8e1517267b71b7bd6849edb152 WatchSource:0}: Error finding container f5d7cb0236739c8c9a65c274dd95b566e5575c8e1517267b71b7bd6849edb152: Status 404 returned error can't find the container with id f5d7cb0236739c8c9a65c274dd95b566e5575c8e1517267b71b7bd6849edb152
Oct 31 04:15:01.778025 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.777986    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf" event={"ID":"8b45f245-d9e1-4476-bdb9-4a840c078444","Type":"ContainerStarted","Data":"b0618210ace9c7114e4016c1a84216d85e8c25084b69788754ecf65c714a7863"}
Oct 31 04:15:01.778225 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.778049    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf" event={"ID":"8b45f245-d9e1-4476-bdb9-4a840c078444","Type":"ContainerStarted","Data":"adfe3614649534f9ec655ee230dad1daa713aa50ee4366ae3512b249a9ffa066"}
Oct 31 04:15:01.779710 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.779680    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw" event={"ID":"ecfcb4c4-007c-4acc-a20c-092b12cfc44b","Type":"ContainerStarted","Data":"a40a2ec54ad1623893f633341259b7ec981a937cce86c5f6f4c6a249cde79cb8"}
Oct 31 04:15:01.779871 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.779735    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw" event={"ID":"ecfcb4c4-007c-4acc-a20c-092b12cfc44b","Type":"ContainerStarted","Data":"25465c427dd2a77642169114bdd26e6a02997e86dd2547748c5459319e6c49af"}
Oct 31 04:15:01.781587 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.781562    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf" event={"ID":"713a4870-e144-4960-bbff-0c4e623ba57c","Type":"ContainerStarted","Data":"f75f6554af61c1bce82045b37846d2ad637fb308a0e019808a13f4d79f02e1d4"}
Oct 31 04:15:01.782013 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.781606    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf" event={"ID":"713a4870-e144-4960-bbff-0c4e623ba57c","Type":"ContainerStarted","Data":"f5d7cb0236739c8c9a65c274dd95b566e5575c8e1517267b71b7bd6849edb152"}
Oct 31 04:15:01.783287 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.783265    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw" event={"ID":"64a9e5b0-9d34-468d-a70a-304974436a9b","Type":"ContainerStarted","Data":"95c608fcad3acb2f4d1c9b1caa9bc60cda725d3e8ece95956c4040e5a3c17f1a"}
Oct 31 04:15:01.783388 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.783306    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw" event={"ID":"64a9e5b0-9d34-468d-a70a-304974436a9b","Type":"ContainerStarted","Data":"26a469e506bfc0145dee2d6eb033c65afde8dbf80b26651ac0884fbe852c6863"}
Oct 31 04:15:01.784837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.784810    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb" event={"ID":"8797d751-7c89-4eb8-aaf9-abf0c81efcca","Type":"ContainerStarted","Data":"0c92454e4c23f512cfd0a15a1bbb5bf1291f10edd6c2de141c28a0a2f205b415"}
Oct 31 04:15:01.784942 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.784857    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb" event={"ID":"8797d751-7c89-4eb8-aaf9-abf0c81efcca","Type":"ContainerStarted","Data":"40ccf3abfe910cabbbd0c2aab68a278c7dc6f60bbec0ef9c10fcc9eeab029564"}
Oct 31 04:15:01.786377 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.786355    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm" event={"ID":"e4447ca2-888d-4950-b75d-b6149d71791a","Type":"ContainerStarted","Data":"e6d5388d6d90342764cea5a91f6bcea664e017a1dec160e681cddb4e1a95f506"}
Oct 31 04:15:01.786485 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.786395    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm" event={"ID":"e4447ca2-888d-4950-b75d-b6149d71791a","Type":"ContainerStarted","Data":"66e7ecf02ab2f3133bf24ed53fc9cb27d40271e5cb7eb08caa9ce5693502ed42"}
Oct 31 04:15:01.787774 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.787756    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r" event={"ID":"a2146db6-ed0b-4776-930f-e1f915526586","Type":"ContainerStarted","Data":"0962ac992afa2033f3d74ba4c1364c6d39c4556ed5e00e045772cf8e92f650e9"}
Oct 31 04:15:01.787939 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.787799    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r" event={"ID":"a2146db6-ed0b-4776-930f-e1f915526586","Type":"ContainerStarted","Data":"94c0135df4a746d2e7daf5a086458c8283d3f5695edd3f314c70fefb6688d847"}
Oct 31 04:15:01.892357 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:01.892298    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf" podStartSLOduration=1.8922833209999999 podStartE2EDuration="1.892283321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:01.88987901 +0000 UTC m=+4054.856212351" watchObservedRunningTime="2026-10-31 04:15:01.892283321 +0000 UTC m=+4054.858616665"
Oct 31 04:15:02.007469 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.007414    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf" podStartSLOduration=2.007399118 podStartE2EDuration="2.007399118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:02.004384933 +0000 UTC m=+4054.970718277" watchObservedRunningTime="2026-10-31 04:15:02.007399118 +0000 UTC m=+4054.973732462"
Oct 31 04:15:02.119141 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.119034    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r" podStartSLOduration=2.119020669 podStartE2EDuration="2.119020669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:02.118291697 +0000 UTC m=+4055.084625041" watchObservedRunningTime="2026-10-31 04:15:02.119020669 +0000 UTC m=+4055.085354012"
Oct 31 04:15:02.232946 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.232894    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb" podStartSLOduration=2.23287831 podStartE2EDuration="2.23287831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:02.23022336 +0000 UTC m=+4055.196556705" watchObservedRunningTime="2026-10-31 04:15:02.23287831 +0000 UTC m=+4055.199211688"
Oct 31 04:15:02.344446 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.344393    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm" podStartSLOduration=2.344378535 podStartE2EDuration="2.344378535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:02.34280353 +0000 UTC m=+4055.309136871" watchObservedRunningTime="2026-10-31 04:15:02.344378535 +0000 UTC m=+4055.310711929"
Oct 31 04:15:02.455368 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.455281    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw" podStartSLOduration=2.455263018 podStartE2EDuration="2.455263018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:02.453726017 +0000 UTC m=+4055.420059361" watchObservedRunningTime="2026-10-31 04:15:02.455263018 +0000 UTC m=+4055.421596363"
Oct 31 04:15:02.569288 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.569222    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw" podStartSLOduration=2.5692058859999998 podStartE2EDuration="2.569205886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:02.56566605 +0000 UTC m=+4055.531999395" watchObservedRunningTime="2026-10-31 04:15:02.569205886 +0000 UTC m=+4055.535539230"
Oct 31 04:15:02.802121 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.802075    4900 generic.go:356] "Generic (PLEG): container finished" podID="8b45f245-d9e1-4476-bdb9-4a840c078444" containerID="b0618210ace9c7114e4016c1a84216d85e8c25084b69788754ecf65c714a7863" exitCode=7
Oct 31 04:15:02.802409 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.802232    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf" event={"ID":"8b45f245-d9e1-4476-bdb9-4a840c078444","Type":"ContainerDied","Data":"b0618210ace9c7114e4016c1a84216d85e8c25084b69788754ecf65c714a7863"}
Oct 31 04:15:02.802856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.802834    4900 scope.go:122] "RemoveContainer" containerID="b0618210ace9c7114e4016c1a84216d85e8c25084b69788754ecf65c714a7863"
Oct 31 04:15:02.804947 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.804922    4900 generic.go:356] "Generic (PLEG): container finished" podID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b" containerID="a40a2ec54ad1623893f633341259b7ec981a937cce86c5f6f4c6a249cde79cb8" exitCode=60
Oct 31 04:15:02.805062 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.805045    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw" event={"ID":"ecfcb4c4-007c-4acc-a20c-092b12cfc44b","Type":"ContainerDied","Data":"a40a2ec54ad1623893f633341259b7ec981a937cce86c5f6f4c6a249cde79cb8"}
Oct 31 04:15:02.805511 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.805490    4900 scope.go:122] "RemoveContainer" containerID="a40a2ec54ad1623893f633341259b7ec981a937cce86c5f6f4c6a249cde79cb8"
Oct 31 04:15:02.808291 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.808111    4900 generic.go:356] "Generic (PLEG): container finished" podID="713a4870-e144-4960-bbff-0c4e623ba57c" containerID="f75f6554af61c1bce82045b37846d2ad637fb308a0e019808a13f4d79f02e1d4" exitCode=60
Oct 31 04:15:02.808291 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.808245    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf" event={"ID":"713a4870-e144-4960-bbff-0c4e623ba57c","Type":"ContainerDied","Data":"f75f6554af61c1bce82045b37846d2ad637fb308a0e019808a13f4d79f02e1d4"}
Oct 31 04:15:02.808501 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.808420    4900 scope.go:122] "RemoveContainer" containerID="f75f6554af61c1bce82045b37846d2ad637fb308a0e019808a13f4d79f02e1d4"
Oct 31 04:15:02.810306 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.810121    4900 generic.go:356] "Generic (PLEG): container finished" podID="64a9e5b0-9d34-468d-a70a-304974436a9b" containerID="95c608fcad3acb2f4d1c9b1caa9bc60cda725d3e8ece95956c4040e5a3c17f1a" exitCode=60
Oct 31 04:15:02.810306 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.810198    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw" event={"ID":"64a9e5b0-9d34-468d-a70a-304974436a9b","Type":"ContainerDied","Data":"95c608fcad3acb2f4d1c9b1caa9bc60cda725d3e8ece95956c4040e5a3c17f1a"}
Oct 31 04:15:02.810491 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.810429    4900 scope.go:122] "RemoveContainer" containerID="95c608fcad3acb2f4d1c9b1caa9bc60cda725d3e8ece95956c4040e5a3c17f1a"
Oct 31 04:15:02.811926 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.811901    4900 generic.go:356] "Generic (PLEG): container finished" podID="8797d751-7c89-4eb8-aaf9-abf0c81efcca" containerID="0c92454e4c23f512cfd0a15a1bbb5bf1291f10edd6c2de141c28a0a2f205b415" exitCode=60
Oct 31 04:15:02.812031 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.811971    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb" event={"ID":"8797d751-7c89-4eb8-aaf9-abf0c81efcca","Type":"ContainerDied","Data":"0c92454e4c23f512cfd0a15a1bbb5bf1291f10edd6c2de141c28a0a2f205b415"}
Oct 31 04:15:02.812172 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.812157    4900 scope.go:122] "RemoveContainer" containerID="0c92454e4c23f512cfd0a15a1bbb5bf1291f10edd6c2de141c28a0a2f205b415"
Oct 31 04:15:02.813708 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.813682    4900 generic.go:356] "Generic (PLEG): container finished" podID="e4447ca2-888d-4950-b75d-b6149d71791a" containerID="e6d5388d6d90342764cea5a91f6bcea664e017a1dec160e681cddb4e1a95f506" exitCode=60
Oct 31 04:15:02.813832 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.813759    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm" event={"ID":"e4447ca2-888d-4950-b75d-b6149d71791a","Type":"ContainerDied","Data":"e6d5388d6d90342764cea5a91f6bcea664e017a1dec160e681cddb4e1a95f506"}
Oct 31 04:15:02.814008 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.813993    4900 scope.go:122] "RemoveContainer" containerID="e6d5388d6d90342764cea5a91f6bcea664e017a1dec160e681cddb4e1a95f506"
Oct 31 04:15:02.816124 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.816107    4900 generic.go:356] "Generic (PLEG): container finished" podID="a2146db6-ed0b-4776-930f-e1f915526586" containerID="0962ac992afa2033f3d74ba4c1364c6d39c4556ed5e00e045772cf8e92f650e9" exitCode=60
Oct 31 04:15:02.816187 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.816161    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r" event={"ID":"a2146db6-ed0b-4776-930f-e1f915526586","Type":"ContainerDied","Data":"0962ac992afa2033f3d74ba4c1364c6d39c4556ed5e00e045772cf8e92f650e9"}
Oct 31 04:15:02.816419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:02.816404    4900 scope.go:122] "RemoveContainer" containerID="0962ac992afa2033f3d74ba4c1364c6d39c4556ed5e00e045772cf8e92f650e9"
Oct 31 04:15:03.825421 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:03.825386    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf" event={"ID":"8b45f245-d9e1-4476-bdb9-4a840c078444","Type":"ContainerStarted","Data":"fd3b16f1ba804b2c58adc409d1010d171337eb6ede86dbd67b254f919b642934"}
Oct 31 04:15:03.827734 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:03.827703    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw" event={"ID":"ecfcb4c4-007c-4acc-a20c-092b12cfc44b","Type":"ContainerStarted","Data":"1d8ae56c4e68e1affc0f2d7f1aaa78e60d0f390322f3cab054ff4985f517112c"}
Oct 31 04:15:03.830002 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:03.829974    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf" event={"ID":"713a4870-e144-4960-bbff-0c4e623ba57c","Type":"ContainerStarted","Data":"6e5e803cd7fa2b4e093879f2d78f1da9cdf83e6b0f453171090621bf821d58d4"}
Oct 31 04:15:03.832017 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:03.831992    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw" event={"ID":"64a9e5b0-9d34-468d-a70a-304974436a9b","Type":"ContainerStarted","Data":"45a93d0fcf825bae614a2f3eb7e9e70fadc45e44a30386c60b2da6598a785203"}
Oct 31 04:15:03.839989 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:03.833922    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb" event={"ID":"8797d751-7c89-4eb8-aaf9-abf0c81efcca","Type":"ContainerStarted","Data":"ec4c1b40f6de3b08bc1fbdb687897d199528c98f5614b2dad6d1f5594659e37a"}
Oct 31 04:15:03.845527 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:03.845489    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm" event={"ID":"e4447ca2-888d-4950-b75d-b6149d71791a","Type":"ContainerStarted","Data":"b0ce0921c080ed11887c4d461baae2dbf2a915d10d0695f09fec803a91a7dedb"}
Oct 31 04:15:03.848704 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:03.848669    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r" event={"ID":"a2146db6-ed0b-4776-930f-e1f915526586","Type":"ContainerStarted","Data":"d5286444254ade7e93341f1826470392df104f81351a83e6c1200d999305b3ff"}
Oct 31 04:15:03.980146 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:03.980113    4900 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8797d751_7c89_4eb8_aaf9_abf0c81efcca.slice/crio-conmon-ec4c1b40f6de3b08bc1fbdb687897d199528c98f5614b2dad6d1f5594659e37a.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 04:15:04.869749 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.869707    4900 generic.go:356] "Generic (PLEG): container finished" podID="a2146db6-ed0b-4776-930f-e1f915526586" containerID="d5286444254ade7e93341f1826470392df104f81351a83e6c1200d999305b3ff" exitCode=60
Oct 31 04:15:04.870230 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.869830    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r" event={"ID":"a2146db6-ed0b-4776-930f-e1f915526586","Type":"ContainerDied","Data":"d5286444254ade7e93341f1826470392df104f81351a83e6c1200d999305b3ff"}
Oct 31 04:15:04.870230 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.869933    4900 scope.go:122] "RemoveContainer" containerID="0962ac992afa2033f3d74ba4c1364c6d39c4556ed5e00e045772cf8e92f650e9"
Oct 31 04:15:04.870230 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.870097    4900 scope.go:122] "RemoveContainer" containerID="d5286444254ade7e93341f1826470392df104f81351a83e6c1200d999305b3ff"
Oct 31 04:15:04.870472 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:04.870447    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r_odh-ai-gateway-infra(a2146db6-ed0b-4776-930f-e1f915526586)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r" podUID="a2146db6-ed0b-4776-930f-e1f915526586"
Oct 31 04:15:04.872589 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.872565    4900 generic.go:356] "Generic (PLEG): container finished" podID="8b45f245-d9e1-4476-bdb9-4a840c078444" containerID="fd3b16f1ba804b2c58adc409d1010d171337eb6ede86dbd67b254f919b642934" exitCode=60
Oct 31 04:15:04.872722 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.872638    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf" event={"ID":"8b45f245-d9e1-4476-bdb9-4a840c078444","Type":"ContainerDied","Data":"fd3b16f1ba804b2c58adc409d1010d171337eb6ede86dbd67b254f919b642934"}
Oct 31 04:15:04.872983 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.872968    4900 scope.go:122] "RemoveContainer" containerID="fd3b16f1ba804b2c58adc409d1010d171337eb6ede86dbd67b254f919b642934"
Oct 31 04:15:04.873401 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:04.873377    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf_odh-ai-gateway-infra(8b45f245-d9e1-4476-bdb9-4a840c078444)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf" podUID="8b45f245-d9e1-4476-bdb9-4a840c078444"
Oct 31 04:15:04.875713 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.875692    4900 generic.go:356] "Generic (PLEG): container finished" podID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b" containerID="1d8ae56c4e68e1affc0f2d7f1aaa78e60d0f390322f3cab054ff4985f517112c" exitCode=60
Oct 31 04:15:04.875882 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.875858    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw" event={"ID":"ecfcb4c4-007c-4acc-a20c-092b12cfc44b","Type":"ContainerDied","Data":"1d8ae56c4e68e1affc0f2d7f1aaa78e60d0f390322f3cab054ff4985f517112c"}
Oct 31 04:15:04.876120 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.876106    4900 scope.go:122] "RemoveContainer" containerID="1d8ae56c4e68e1affc0f2d7f1aaa78e60d0f390322f3cab054ff4985f517112c"
Oct 31 04:15:04.876435 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:04.876412    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw_odh-ai-gateway-infra(ecfcb4c4-007c-4acc-a20c-092b12cfc44b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw" podUID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b"
Oct 31 04:15:04.878344 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.878319    4900 generic.go:356] "Generic (PLEG): container finished" podID="713a4870-e144-4960-bbff-0c4e623ba57c" containerID="6e5e803cd7fa2b4e093879f2d78f1da9cdf83e6b0f453171090621bf821d58d4" exitCode=60
Oct 31 04:15:04.878446 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.878397    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf" event={"ID":"713a4870-e144-4960-bbff-0c4e623ba57c","Type":"ContainerDied","Data":"6e5e803cd7fa2b4e093879f2d78f1da9cdf83e6b0f453171090621bf821d58d4"}
Oct 31 04:15:04.878669 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.878645    4900 scope.go:122] "RemoveContainer" containerID="6e5e803cd7fa2b4e093879f2d78f1da9cdf83e6b0f453171090621bf821d58d4"
Oct 31 04:15:04.879056 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:04.879029    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf_odh-ai-gateway-infra(713a4870-e144-4960-bbff-0c4e623ba57c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf" podUID="713a4870-e144-4960-bbff-0c4e623ba57c"
Oct 31 04:15:04.880720 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.880700    4900 generic.go:356] "Generic (PLEG): container finished" podID="64a9e5b0-9d34-468d-a70a-304974436a9b" containerID="45a93d0fcf825bae614a2f3eb7e9e70fadc45e44a30386c60b2da6598a785203" exitCode=60
Oct 31 04:15:04.880865 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.880822    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw" event={"ID":"64a9e5b0-9d34-468d-a70a-304974436a9b","Type":"ContainerDied","Data":"45a93d0fcf825bae614a2f3eb7e9e70fadc45e44a30386c60b2da6598a785203"}
Oct 31 04:15:04.881060 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.881043    4900 scope.go:122] "RemoveContainer" containerID="45a93d0fcf825bae614a2f3eb7e9e70fadc45e44a30386c60b2da6598a785203"
Oct 31 04:15:04.881382 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:04.881360    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw_odh-ai-gateway-infra(64a9e5b0-9d34-468d-a70a-304974436a9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw" podUID="64a9e5b0-9d34-468d-a70a-304974436a9b"
Oct 31 04:15:04.882750 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.882730    4900 generic.go:356] "Generic (PLEG): container finished" podID="8797d751-7c89-4eb8-aaf9-abf0c81efcca" containerID="ec4c1b40f6de3b08bc1fbdb687897d199528c98f5614b2dad6d1f5594659e37a" exitCode=60
Oct 31 04:15:04.882865 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.882806    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb" event={"ID":"8797d751-7c89-4eb8-aaf9-abf0c81efcca","Type":"ContainerDied","Data":"ec4c1b40f6de3b08bc1fbdb687897d199528c98f5614b2dad6d1f5594659e37a"}
Oct 31 04:15:04.883095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.883077    4900 scope.go:122] "RemoveContainer" containerID="ec4c1b40f6de3b08bc1fbdb687897d199528c98f5614b2dad6d1f5594659e37a"
Oct 31 04:15:04.883363 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:04.883342    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb_odh-ai-gateway-infra(8797d751-7c89-4eb8-aaf9-abf0c81efcca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb" podUID="8797d751-7c89-4eb8-aaf9-abf0c81efcca"
Oct 31 04:15:04.884936 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.884913    4900 generic.go:356] "Generic (PLEG): container finished" podID="e4447ca2-888d-4950-b75d-b6149d71791a" containerID="b0ce0921c080ed11887c4d461baae2dbf2a915d10d0695f09fec803a91a7dedb" exitCode=60
Oct 31 04:15:04.885021 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.884965    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm" event={"ID":"e4447ca2-888d-4950-b75d-b6149d71791a","Type":"ContainerDied","Data":"b0ce0921c080ed11887c4d461baae2dbf2a915d10d0695f09fec803a91a7dedb"}
Oct 31 04:15:04.885196 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.885183    4900 scope.go:122] "RemoveContainer" containerID="b0ce0921c080ed11887c4d461baae2dbf2a915d10d0695f09fec803a91a7dedb"
Oct 31 04:15:04.885405 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:04.885389    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm_odh-ai-gateway-infra(e4447ca2-888d-4950-b75d-b6149d71791a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm" podUID="e4447ca2-888d-4950-b75d-b6149d71791a"
Oct 31 04:15:04.886247 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.886221    4900 scope.go:122] "RemoveContainer" containerID="b0618210ace9c7114e4016c1a84216d85e8c25084b69788754ecf65c714a7863"
Oct 31 04:15:04.900177 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.900147    4900 scope.go:122] "RemoveContainer" containerID="a40a2ec54ad1623893f633341259b7ec981a937cce86c5f6f4c6a249cde79cb8"
Oct 31 04:15:04.912922 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.912892    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:04.922159 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.922133    4900 scope.go:122] "RemoveContainer" containerID="f75f6554af61c1bce82045b37846d2ad637fb308a0e019808a13f4d79f02e1d4"
Oct 31 04:15:04.940009 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.939988    4900 scope.go:122] "RemoveContainer" containerID="95c608fcad3acb2f4d1c9b1caa9bc60cda725d3e8ece95956c4040e5a3c17f1a"
Oct 31 04:15:04.953823 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.953772    4900 scope.go:122] "RemoveContainer" containerID="0c92454e4c23f512cfd0a15a1bbb5bf1291f10edd6c2de141c28a0a2f205b415"
Oct 31 04:15:04.967114 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:04.967087    4900 scope.go:122] "RemoveContainer" containerID="e6d5388d6d90342764cea5a91f6bcea664e017a1dec160e681cddb4e1a95f506"
Oct 31 04:15:05.322047 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:05.322012    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:05.893973 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:05.893945    4900 scope.go:122] "RemoveContainer" containerID="fd3b16f1ba804b2c58adc409d1010d171337eb6ede86dbd67b254f919b642934"
Oct 31 04:15:05.894392 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:05.894296    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf_odh-ai-gateway-infra(8b45f245-d9e1-4476-bdb9-4a840c078444)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf" podUID="8b45f245-d9e1-4476-bdb9-4a840c078444"
Oct 31 04:15:05.895459 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:05.895439    4900 scope.go:122] "RemoveContainer" containerID="1d8ae56c4e68e1affc0f2d7f1aaa78e60d0f390322f3cab054ff4985f517112c"
Oct 31 04:15:05.895748 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:05.895723    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw_odh-ai-gateway-infra(ecfcb4c4-007c-4acc-a20c-092b12cfc44b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw" podUID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b"
Oct 31 04:15:05.896891 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:05.896871    4900 scope.go:122] "RemoveContainer" containerID="6e5e803cd7fa2b4e093879f2d78f1da9cdf83e6b0f453171090621bf821d58d4"
Oct 31 04:15:05.897106 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:05.897082    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf_odh-ai-gateway-infra(713a4870-e144-4960-bbff-0c4e623ba57c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf" podUID="713a4870-e144-4960-bbff-0c4e623ba57c"
Oct 31 04:15:05.898334 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:05.898315    4900 scope.go:122] "RemoveContainer" containerID="45a93d0fcf825bae614a2f3eb7e9e70fadc45e44a30386c60b2da6598a785203"
Oct 31 04:15:05.898550 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:05.898531    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw_odh-ai-gateway-infra(64a9e5b0-9d34-468d-a70a-304974436a9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw" podUID="64a9e5b0-9d34-468d-a70a-304974436a9b"
Oct 31 04:15:05.899709 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:05.899684    4900 scope.go:122] "RemoveContainer" containerID="ec4c1b40f6de3b08bc1fbdb687897d199528c98f5614b2dad6d1f5594659e37a"
Oct 31 04:15:05.899939 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:05.899921    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb_odh-ai-gateway-infra(8797d751-7c89-4eb8-aaf9-abf0c81efcca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb" podUID="8797d751-7c89-4eb8-aaf9-abf0c81efcca"
Oct 31 04:15:05.901114 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:05.901098    4900 scope.go:122] "RemoveContainer" containerID="b0ce0921c080ed11887c4d461baae2dbf2a915d10d0695f09fec803a91a7dedb"
Oct 31 04:15:05.901291 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:05.901267    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm_odh-ai-gateway-infra(e4447ca2-888d-4950-b75d-b6149d71791a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm" podUID="e4447ca2-888d-4950-b75d-b6149d71791a"
Oct 31 04:15:05.902516 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:05.902501    4900 scope.go:122] "RemoveContainer" containerID="d5286444254ade7e93341f1826470392df104f81351a83e6c1200d999305b3ff"
Oct 31 04:15:05.902680 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:05.902665    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r_odh-ai-gateway-infra(a2146db6-ed0b-4776-930f-e1f915526586)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r" podUID="a2146db6-ed0b-4776-930f-e1f915526586"
Oct 31 04:15:06.987959 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:06.987924    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:15:09.771499 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:09.771411    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:14.278544 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.278501    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"]
Oct 31 04:15:14.283153 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.283131    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:14.367805 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.365351    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"]
Oct 31 04:15:14.382894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.382856    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e679fe-serving-cert\""
Oct 31 04:15:14.410625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.410584    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5fe713-f05f-4052-8621-ac82812d0000-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:14.410848 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.410820    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-metrics-certs\") pod \"maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:14.410949 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.410934    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-maas-api-tls\") pod \"maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:14.410996 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.410982    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fpcz\" (UniqueName: \"kubernetes.io/projected/2e5fe713-f05f-4052-8621-ac82812d0000-kube-api-access-8fpcz\") pod \"maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:14.512068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.512028    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-metrics-certs\") pod \"maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:14.512245 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.512106    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-maas-api-tls\") pod \"maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:14.512245 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.512142    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fpcz\" (UniqueName: \"kubernetes.io/projected/2e5fe713-f05f-4052-8621-ac82812d0000-kube-api-access-8fpcz\") pod \"maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:14.512245 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.512192    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5fe713-f05f-4052-8621-ac82812d0000-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:14.512394 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:14.512373    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-serving-cert: secret "maas-api-e2e-inf-b-e679fe-serving-cert" not found
Oct 31 04:15:14.512483 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:14.512470    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-maas-api-tls podName:2e5fe713-f05f-4052-8621-ac82812d0000 nodeName:}" failed. No retries permitted until 2026-10-31 04:15:15.012445757 +0000 UTC m=+4067.978779097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-maas-api-tls") pod "maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k" (UID: "2e5fe713-f05f-4052-8621-ac82812d0000") : secret "maas-api-e2e-inf-b-e679fe-serving-cert" not found
Oct 31 04:15:14.513036 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.513009    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5fe713-f05f-4052-8621-ac82812d0000-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:14.514464 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.514444    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-metrics-certs\") pod \"maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:14.568957 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.568856    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:14.571922 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.571898    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fpcz\" (UniqueName: \"kubernetes.io/projected/2e5fe713-f05f-4052-8621-ac82812d0000-kube-api-access-8fpcz\") pod \"maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:14.786542 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:14.786509    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:15.018315 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:15.018280    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-maas-api-tls\") pod \"maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:15.020858 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:15.020832    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-maas-api-tls\") pod \"maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:15.151395 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:15.151360    4900 scope.go:122] "RemoveContainer" containerID="fd3b16f1ba804b2c58adc409d1010d171337eb6ede86dbd67b254f919b642934"
Oct 31 04:15:15.196810 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:15.196761    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:15.615728 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:15:15.615687    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e5fe713_f05f_4052_8621_ac82812d0000.slice/crio-9f2ce87fb10b5e6b706064c159e374d59a2674c24c75ffead5910d0f1e8dd2bc WatchSource:0}: Error finding container 9f2ce87fb10b5e6b706064c159e374d59a2674c24c75ffead5910d0f1e8dd2bc: Status 404 returned error can't find the container with id 9f2ce87fb10b5e6b706064c159e374d59a2674c24c75ffead5910d0f1e8dd2bc
Oct 31 04:15:15.618601 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:15.618458    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"]
Oct 31 04:15:15.981649 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:15.981565    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf" event={"ID":"8b45f245-d9e1-4476-bdb9-4a840c078444","Type":"ContainerStarted","Data":"0db49df8321e09e62c3f79058d8ec179aa433729805b3b8b17e5ba0445672e8e"}
Oct 31 04:15:15.982900 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:15.982874    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k" event={"ID":"2e5fe713-f05f-4052-8621-ac82812d0000","Type":"ContainerStarted","Data":"9f2ce87fb10b5e6b706064c159e374d59a2674c24c75ffead5910d0f1e8dd2bc"}
Oct 31 04:15:16.154024 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:16.146430    4900 scope.go:122] "RemoveContainer" containerID="ec4c1b40f6de3b08bc1fbdb687897d199528c98f5614b2dad6d1f5594659e37a"
Oct 31 04:15:16.154024 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:16.149215    4900 scope.go:122] "RemoveContainer" containerID="45a93d0fcf825bae614a2f3eb7e9e70fadc45e44a30386c60b2da6598a785203"
Oct 31 04:15:16.990705 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:16.990670    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw" event={"ID":"64a9e5b0-9d34-468d-a70a-304974436a9b","Type":"ContainerStarted","Data":"f2d69e8c550e6dd50164b2d804d296c7c548d7b560964ecce080a346fc283554"}
Oct 31 04:15:16.992497 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:16.992471    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb" event={"ID":"8797d751-7c89-4eb8-aaf9-abf0c81efcca","Type":"ContainerStarted","Data":"6bf50222e3a2fd924f43291d03e1f7d94ab559bd5fd2d27d8f51d05b756ca00d"}
Oct 31 04:15:16.994066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:16.994047    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k" event={"ID":"2e5fe713-f05f-4052-8621-ac82812d0000","Type":"ContainerStarted","Data":"20b938ac97bbc2d6a82dd41fddfd49a75b7de8a5afb0b1b219b9c69dfafcedc1"}
Oct 31 04:15:16.994174 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:16.994117    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:16.995913 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:16.995891    4900 generic.go:356] "Generic (PLEG): container finished" podID="8b45f245-d9e1-4476-bdb9-4a840c078444" containerID="0db49df8321e09e62c3f79058d8ec179aa433729805b3b8b17e5ba0445672e8e" exitCode=60
Oct 31 04:15:16.996022 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:16.995961    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf" event={"ID":"8b45f245-d9e1-4476-bdb9-4a840c078444","Type":"ContainerDied","Data":"0db49df8321e09e62c3f79058d8ec179aa433729805b3b8b17e5ba0445672e8e"}
Oct 31 04:15:16.996022 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:16.996007    4900 scope.go:122] "RemoveContainer" containerID="fd3b16f1ba804b2c58adc409d1010d171337eb6ede86dbd67b254f919b642934"
Oct 31 04:15:16.996263 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:16.996246    4900 scope.go:122] "RemoveContainer" containerID="0db49df8321e09e62c3f79058d8ec179aa433729805b3b8b17e5ba0445672e8e"
Oct 31 04:15:16.996478 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:16.996458    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf_odh-ai-gateway-infra(8b45f245-d9e1-4476-bdb9-4a840c078444)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf" podUID="8b45f245-d9e1-4476-bdb9-4a840c078444"
Oct 31 04:15:17.108743 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:17.108465    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"]
Oct 31 04:15:17.163855 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:17.159916    4900 scope.go:122] "RemoveContainer" containerID="6e5e803cd7fa2b4e093879f2d78f1da9cdf83e6b0f453171090621bf821d58d4"
Oct 31 04:15:17.166833 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:17.166781    4900 scope.go:122] "RemoveContainer" containerID="b0ce0921c080ed11887c4d461baae2dbf2a915d10d0695f09fec803a91a7dedb"
Oct 31 04:15:17.381319 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:17.381262    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k" podStartSLOduration=2.443808078 podStartE2EDuration="3.381248766s" totalImagesPullingTime="937.440688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:17.265336508 +0000 UTC m=+4070.231669878" watchObservedRunningTime="2026-10-31 04:15:17.381248766 +0000 UTC m=+4070.347582148"
Oct 31 04:15:18.005991 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.005955    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf" event={"ID":"713a4870-e144-4960-bbff-0c4e623ba57c","Type":"ContainerStarted","Data":"4222337db3f10d4ce86b971c8d7c0fff9e9e494b2f3906ca749e5e0b059f94f5"}
Oct 31 04:15:18.007867 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.007839    4900 generic.go:356] "Generic (PLEG): container finished" podID="64a9e5b0-9d34-468d-a70a-304974436a9b" containerID="f2d69e8c550e6dd50164b2d804d296c7c548d7b560964ecce080a346fc283554" exitCode=60
Oct 31 04:15:18.007995 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.007908    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw" event={"ID":"64a9e5b0-9d34-468d-a70a-304974436a9b","Type":"ContainerDied","Data":"f2d69e8c550e6dd50164b2d804d296c7c548d7b560964ecce080a346fc283554"}
Oct 31 04:15:18.007995 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.007967    4900 scope.go:122] "RemoveContainer" containerID="45a93d0fcf825bae614a2f3eb7e9e70fadc45e44a30386c60b2da6598a785203"
Oct 31 04:15:18.008365 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.008345    4900 scope.go:122] "RemoveContainer" containerID="f2d69e8c550e6dd50164b2d804d296c7c548d7b560964ecce080a346fc283554"
Oct 31 04:15:18.008713 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:18.008681    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw_odh-ai-gateway-infra(64a9e5b0-9d34-468d-a70a-304974436a9b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw" podUID="64a9e5b0-9d34-468d-a70a-304974436a9b"
Oct 31 04:15:18.009944 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.009925    4900 generic.go:356] "Generic (PLEG): container finished" podID="8797d751-7c89-4eb8-aaf9-abf0c81efcca" containerID="6bf50222e3a2fd924f43291d03e1f7d94ab559bd5fd2d27d8f51d05b756ca00d" exitCode=60
Oct 31 04:15:18.010034 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.009987    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb" event={"ID":"8797d751-7c89-4eb8-aaf9-abf0c81efcca","Type":"ContainerDied","Data":"6bf50222e3a2fd924f43291d03e1f7d94ab559bd5fd2d27d8f51d05b756ca00d"}
Oct 31 04:15:18.010311 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.010297    4900 scope.go:122] "RemoveContainer" containerID="6bf50222e3a2fd924f43291d03e1f7d94ab559bd5fd2d27d8f51d05b756ca00d"
Oct 31 04:15:18.010573 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:18.010557    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb_odh-ai-gateway-infra(8797d751-7c89-4eb8-aaf9-abf0c81efcca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb" podUID="8797d751-7c89-4eb8-aaf9-abf0c81efcca"
Oct 31 04:15:18.011837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.011817    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm" event={"ID":"e4447ca2-888d-4950-b75d-b6149d71791a","Type":"ContainerStarted","Data":"065a6c5ed8bf8fdf22183ed1203465f9c8377cb12a34158b4f0d6b3d609cf1d5"}
Oct 31 04:15:18.025657 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.025611    4900 scope.go:122] "RemoveContainer" containerID="ec4c1b40f6de3b08bc1fbdb687897d199528c98f5614b2dad6d1f5594659e37a"
Oct 31 04:15:18.153817 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.146308    4900 scope.go:122] "RemoveContainer" containerID="d5286444254ade7e93341f1826470392df104f81351a83e6c1200d999305b3ff"
Oct 31 04:15:18.166827 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.162590    4900 scope.go:122] "RemoveContainer" containerID="1d8ae56c4e68e1affc0f2d7f1aaa78e60d0f390322f3cab054ff4985f517112c"
Oct 31 04:15:18.172658 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.171271    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"]
Oct 31 04:15:18.267720 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.267654    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"
Oct 31 04:15:18.390905 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.390872    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"]
Oct 31 04:15:18.394900 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.394879    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b45f245-d9e1-4476-bdb9-4a840c078444-kube-api-access-psd5n\" (UniqueName: \"kubernetes.io/projected/8b45f245-d9e1-4476-bdb9-4a840c078444-kube-api-access-psd5n\") pod \"8b45f245-d9e1-4476-bdb9-4a840c078444\" (UID: \"8b45f245-d9e1-4476-bdb9-4a840c078444\") "
Oct 31 04:15:18.395090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.395078    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b45f245-d9e1-4476-bdb9-4a840c078444-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b45f245-d9e1-4476-bdb9-4a840c078444-maas-api-service-ca\") pod \"8b45f245-d9e1-4476-bdb9-4a840c078444\" (UID: \"8b45f245-d9e1-4476-bdb9-4a840c078444\") "
Oct 31 04:15:18.395423 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.395400    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b45f245-d9e1-4476-bdb9-4a840c078444-maas-api-service-ca" pod "8b45f245-d9e1-4476-bdb9-4a840c078444" (UID: "8b45f245-d9e1-4476-bdb9-4a840c078444"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:18.395634 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.395618    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b45f245-d9e1-4476-bdb9-4a840c078444-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:18.396807 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.396770    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b45f245-d9e1-4476-bdb9-4a840c078444-kube-api-access-psd5n" pod "8b45f245-d9e1-4476-bdb9-4a840c078444" (UID: "8b45f245-d9e1-4476-bdb9-4a840c078444"). InnerVolumeSpecName "kube-api-access-psd5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:18.496755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:18.496723    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psd5n\" (UniqueName: \"kubernetes.io/projected/8b45f245-d9e1-4476-bdb9-4a840c078444-kube-api-access-psd5n\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:19.020486 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.020457    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"
Oct 31 04:15:19.020990 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.020460    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf" event={"ID":"8b45f245-d9e1-4476-bdb9-4a840c078444","Type":"ContainerDied","Data":"adfe3614649534f9ec655ee230dad1daa713aa50ee4366ae3512b249a9ffa066"}
Oct 31 04:15:19.021095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.021076    4900 scope.go:122] "RemoveContainer" containerID="0db49df8321e09e62c3f79058d8ec179aa433729805b3b8b17e5ba0445672e8e"
Oct 31 04:15:19.023081 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.023059    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw" event={"ID":"ecfcb4c4-007c-4acc-a20c-092b12cfc44b","Type":"ContainerStarted","Data":"b01d9ab50569dd2e8dfae5188f6ff0225c67e5a1ee67ff2c06e6b625ccaf810a"}
Oct 31 04:15:19.025253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.025226    4900 generic.go:356] "Generic (PLEG): container finished" podID="713a4870-e144-4960-bbff-0c4e623ba57c" containerID="4222337db3f10d4ce86b971c8d7c0fff9e9e494b2f3906ca749e5e0b059f94f5" exitCode=60
Oct 31 04:15:19.025343 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.025294    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf" event={"ID":"713a4870-e144-4960-bbff-0c4e623ba57c","Type":"ContainerDied","Data":"4222337db3f10d4ce86b971c8d7c0fff9e9e494b2f3906ca749e5e0b059f94f5"}
Oct 31 04:15:19.025693 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.025674    4900 scope.go:122] "RemoveContainer" containerID="4222337db3f10d4ce86b971c8d7c0fff9e9e494b2f3906ca749e5e0b059f94f5"
Oct 31 04:15:19.026038 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:19.026006    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf_odh-ai-gateway-infra(713a4870-e144-4960-bbff-0c4e623ba57c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf" podUID="713a4870-e144-4960-bbff-0c4e623ba57c"
Oct 31 04:15:19.033413 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.033392    4900 scope.go:122] "RemoveContainer" containerID="6e5e803cd7fa2b4e093879f2d78f1da9cdf83e6b0f453171090621bf821d58d4"
Oct 31 04:15:19.069977 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.069938    4900 generic.go:356] "Generic (PLEG): container finished" podID="e4447ca2-888d-4950-b75d-b6149d71791a" containerID="065a6c5ed8bf8fdf22183ed1203465f9c8377cb12a34158b4f0d6b3d609cf1d5" exitCode=60
Oct 31 04:15:19.070138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.070079    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm" event={"ID":"e4447ca2-888d-4950-b75d-b6149d71791a","Type":"ContainerDied","Data":"065a6c5ed8bf8fdf22183ed1203465f9c8377cb12a34158b4f0d6b3d609cf1d5"}
Oct 31 04:15:19.070666 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.070636    4900 scope.go:122] "RemoveContainer" containerID="065a6c5ed8bf8fdf22183ed1203465f9c8377cb12a34158b4f0d6b3d609cf1d5"
Oct 31 04:15:19.071101 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:19.071069    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm_odh-ai-gateway-infra(e4447ca2-888d-4950-b75d-b6149d71791a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm" podUID="e4447ca2-888d-4950-b75d-b6149d71791a"
Oct 31 04:15:19.074370 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.074231    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r" event={"ID":"a2146db6-ed0b-4776-930f-e1f915526586","Type":"ContainerStarted","Data":"548af04c1963798c2c0ea1f44b6fe48b229f656f8e19f55281698044bef18a3a"}
Oct 31 04:15:19.087143 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.087117    4900 scope.go:122] "RemoveContainer" containerID="b0ce0921c080ed11887c4d461baae2dbf2a915d10d0695f09fec803a91a7dedb"
Oct 31 04:15:19.245429 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.245396    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"]
Oct 31 04:15:19.261133 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.261108    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"
Oct 31 04:15:19.287929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.287870    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"
Oct 31 04:15:19.311229 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.311202    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8797d751-7c89-4eb8-aaf9-abf0c81efcca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8797d751-7c89-4eb8-aaf9-abf0c81efcca-maas-api-service-ca\") pod \"8797d751-7c89-4eb8-aaf9-abf0c81efcca\" (UID: \"8797d751-7c89-4eb8-aaf9-abf0c81efcca\") "
Oct 31 04:15:19.311455 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.311436    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a9e5b0-9d34-468d-a70a-304974436a9b-kube-api-access-5ndr5\" (UniqueName: \"kubernetes.io/projected/64a9e5b0-9d34-468d-a70a-304974436a9b-kube-api-access-5ndr5\") pod \"64a9e5b0-9d34-468d-a70a-304974436a9b\" (UID: \"64a9e5b0-9d34-468d-a70a-304974436a9b\") "
Oct 31 04:15:19.311528 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.311498    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8797d751-7c89-4eb8-aaf9-abf0c81efcca-kube-api-access-hzwr5\" (UniqueName: \"kubernetes.io/projected/8797d751-7c89-4eb8-aaf9-abf0c81efcca-kube-api-access-hzwr5\") pod \"8797d751-7c89-4eb8-aaf9-abf0c81efcca\" (UID: \"8797d751-7c89-4eb8-aaf9-abf0c81efcca\") "
Oct 31 04:15:19.311590 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.311559    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a9e5b0-9d34-468d-a70a-304974436a9b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a9e5b0-9d34-468d-a70a-304974436a9b-maas-api-service-ca\") pod \"64a9e5b0-9d34-468d-a70a-304974436a9b\" (UID: \"64a9e5b0-9d34-468d-a70a-304974436a9b\") "
Oct 31 04:15:19.311655 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.311603    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8797d751-7c89-4eb8-aaf9-abf0c81efcca-maas-api-service-ca" pod "8797d751-7c89-4eb8-aaf9-abf0c81efcca" (UID: "8797d751-7c89-4eb8-aaf9-abf0c81efcca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:19.311936 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.311915    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a9e5b0-9d34-468d-a70a-304974436a9b-maas-api-service-ca" pod "64a9e5b0-9d34-468d-a70a-304974436a9b" (UID: "64a9e5b0-9d34-468d-a70a-304974436a9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:19.312193 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.312176    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a9e5b0-9d34-468d-a70a-304974436a9b-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:19.312252 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.312194    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8797d751-7c89-4eb8-aaf9-abf0c81efcca-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:19.314067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.314050    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a9e5b0-9d34-468d-a70a-304974436a9b-kube-api-access-5ndr5" pod "64a9e5b0-9d34-468d-a70a-304974436a9b" (UID: "64a9e5b0-9d34-468d-a70a-304974436a9b"). InnerVolumeSpecName "kube-api-access-5ndr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:19.314166 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.314077    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8797d751-7c89-4eb8-aaf9-abf0c81efcca-kube-api-access-hzwr5" pod "8797d751-7c89-4eb8-aaf9-abf0c81efcca" (UID: "8797d751-7c89-4eb8-aaf9-abf0c81efcca"). InnerVolumeSpecName "kube-api-access-hzwr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:19.413778 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.413746    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ndr5\" (UniqueName: \"kubernetes.io/projected/64a9e5b0-9d34-468d-a70a-304974436a9b-kube-api-access-5ndr5\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:19.413778 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.413774    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzwr5\" (UniqueName: \"kubernetes.io/projected/8797d751-7c89-4eb8-aaf9-abf0c81efcca-kube-api-access-hzwr5\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:19.500128 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.500095    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"]
Oct 31 04:15:19.714596 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.714562    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:19.790778 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.790699    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"]
Oct 31 04:15:19.799997 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:19.799968    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-6f30f2-29890335-85scf"]
Oct 31 04:15:20.083020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.082929    4900 generic.go:356] "Generic (PLEG): container finished" podID="a2146db6-ed0b-4776-930f-e1f915526586" containerID="548af04c1963798c2c0ea1f44b6fe48b229f656f8e19f55281698044bef18a3a" exitCode=60
Oct 31 04:15:20.083020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.083003    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r" event={"ID":"a2146db6-ed0b-4776-930f-e1f915526586","Type":"ContainerDied","Data":"548af04c1963798c2c0ea1f44b6fe48b229f656f8e19f55281698044bef18a3a"}
Oct 31 04:15:20.083536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.083103    4900 scope.go:122] "RemoveContainer" containerID="d5286444254ade7e93341f1826470392df104f81351a83e6c1200d999305b3ff"
Oct 31 04:15:20.083536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.083523    4900 scope.go:122] "RemoveContainer" containerID="548af04c1963798c2c0ea1f44b6fe48b229f656f8e19f55281698044bef18a3a"
Oct 31 04:15:20.083935 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:20.083908    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r_odh-ai-gateway-infra(a2146db6-ed0b-4776-930f-e1f915526586)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r" podUID="a2146db6-ed0b-4776-930f-e1f915526586"
Oct 31 04:15:20.086909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.086886    4900 generic.go:356] "Generic (PLEG): container finished" podID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b" containerID="b01d9ab50569dd2e8dfae5188f6ff0225c67e5a1ee67ff2c06e6b625ccaf810a" exitCode=60
Oct 31 04:15:20.087019 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.086960    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw" event={"ID":"ecfcb4c4-007c-4acc-a20c-092b12cfc44b","Type":"ContainerDied","Data":"b01d9ab50569dd2e8dfae5188f6ff0225c67e5a1ee67ff2c06e6b625ccaf810a"}
Oct 31 04:15:20.087379 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.087345    4900 scope.go:122] "RemoveContainer" containerID="b01d9ab50569dd2e8dfae5188f6ff0225c67e5a1ee67ff2c06e6b625ccaf810a"
Oct 31 04:15:20.087709 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:20.087683    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw_odh-ai-gateway-infra(ecfcb4c4-007c-4acc-a20c-092b12cfc44b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw" podUID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b"
Oct 31 04:15:20.090517 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.090492    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw" event={"ID":"64a9e5b0-9d34-468d-a70a-304974436a9b","Type":"ContainerDied","Data":"26a469e506bfc0145dee2d6eb033c65afde8dbf80b26651ac0884fbe852c6863"}
Oct 31 04:15:20.090622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.090497    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"
Oct 31 04:15:20.097160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.094485    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"
Oct 31 04:15:20.097160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.094757    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb" event={"ID":"8797d751-7c89-4eb8-aaf9-abf0c81efcca","Type":"ContainerDied","Data":"40ccf3abfe910cabbbd0c2aab68a278c7dc6f60bbec0ef9c10fcc9eeab029564"}
Oct 31 04:15:20.117340 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.117316    4900 scope.go:122] "RemoveContainer" containerID="1d8ae56c4e68e1affc0f2d7f1aaa78e60d0f390322f3cab054ff4985f517112c"
Oct 31 04:15:20.162919 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.162324    4900 scope.go:122] "RemoveContainer" containerID="f2d69e8c550e6dd50164b2d804d296c7c548d7b560964ecce080a346fc283554"
Oct 31 04:15:20.180860 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.180833    4900 scope.go:122] "RemoveContainer" containerID="6bf50222e3a2fd924f43291d03e1f7d94ab559bd5fd2d27d8f51d05b756ca00d"
Oct 31 04:15:20.252778 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.252748    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"]
Oct 31 04:15:20.256052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.256025    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-e53098-29890335-rqkhw"]
Oct 31 04:15:20.311470 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.311440    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"]
Oct 31 04:15:20.347403 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.347328    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"
Oct 31 04:15:20.349518 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.349498    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"
Oct 31 04:15:20.426412 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.426378    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4447ca2-888d-4950-b75d-b6149d71791a-kube-api-access-b4bbz\" (UniqueName: \"kubernetes.io/projected/e4447ca2-888d-4950-b75d-b6149d71791a-kube-api-access-b4bbz\") pod \"e4447ca2-888d-4950-b75d-b6149d71791a\" (UID: \"e4447ca2-888d-4950-b75d-b6149d71791a\") "
Oct 31 04:15:20.426613 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.426464    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/713a4870-e144-4960-bbff-0c4e623ba57c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713a4870-e144-4960-bbff-0c4e623ba57c-maas-api-service-ca\") pod \"713a4870-e144-4960-bbff-0c4e623ba57c\" (UID: \"713a4870-e144-4960-bbff-0c4e623ba57c\") "
Oct 31 04:15:20.426613 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.426552    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713a4870-e144-4960-bbff-0c4e623ba57c-kube-api-access-pdplk\" (UniqueName: \"kubernetes.io/projected/713a4870-e144-4960-bbff-0c4e623ba57c-kube-api-access-pdplk\") pod \"713a4870-e144-4960-bbff-0c4e623ba57c\" (UID: \"713a4870-e144-4960-bbff-0c4e623ba57c\") "
Oct 31 04:15:20.426746 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.426721    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4447ca2-888d-4950-b75d-b6149d71791a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4447ca2-888d-4950-b75d-b6149d71791a-maas-api-service-ca\") pod \"e4447ca2-888d-4950-b75d-b6149d71791a\" (UID: \"e4447ca2-888d-4950-b75d-b6149d71791a\") "
Oct 31 04:15:20.426894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.426868    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713a4870-e144-4960-bbff-0c4e623ba57c-maas-api-service-ca" pod "713a4870-e144-4960-bbff-0c4e623ba57c" (UID: "713a4870-e144-4960-bbff-0c4e623ba57c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:20.427164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.427138    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4447ca2-888d-4950-b75d-b6149d71791a-maas-api-service-ca" pod "e4447ca2-888d-4950-b75d-b6149d71791a" (UID: "e4447ca2-888d-4950-b75d-b6149d71791a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:20.427946 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.427926    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4447ca2-888d-4950-b75d-b6149d71791a-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:20.428029 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.427953    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/713a4870-e144-4960-bbff-0c4e623ba57c-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:20.429197 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.428900    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713a4870-e144-4960-bbff-0c4e623ba57c-kube-api-access-pdplk" pod "713a4870-e144-4960-bbff-0c4e623ba57c" (UID: "713a4870-e144-4960-bbff-0c4e623ba57c"). InnerVolumeSpecName "kube-api-access-pdplk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:20.430942 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.430914    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4447ca2-888d-4950-b75d-b6149d71791a-kube-api-access-b4bbz" pod "e4447ca2-888d-4950-b75d-b6149d71791a" (UID: "e4447ca2-888d-4950-b75d-b6149d71791a"). InnerVolumeSpecName "kube-api-access-b4bbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:20.529653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.529614    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4bbz\" (UniqueName: \"kubernetes.io/projected/e4447ca2-888d-4950-b75d-b6149d71791a-kube-api-access-b4bbz\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:20.529653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.529650    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdplk\" (UniqueName: \"kubernetes.io/projected/713a4870-e144-4960-bbff-0c4e623ba57c-kube-api-access-pdplk\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:20.590441 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.590408    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"]
Oct 31 04:15:20.593304 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:20.593275    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b59563-29890335-rgpfb"]
Oct 31 04:15:21.037067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.037032    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"]
Oct 31 04:15:21.107728 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.107699    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"
Oct 31 04:15:21.108235 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.107725    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm" event={"ID":"e4447ca2-888d-4950-b75d-b6149d71791a","Type":"ContainerDied","Data":"66e7ecf02ab2f3133bf24ed53fc9cb27d40271e5cb7eb08caa9ce5693502ed42"}
Oct 31 04:15:21.108235 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.107805    4900 scope.go:122] "RemoveContainer" containerID="065a6c5ed8bf8fdf22183ed1203465f9c8377cb12a34158b4f0d6b3d609cf1d5"
Oct 31 04:15:21.115101 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.115080    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"
Oct 31 04:15:21.115225 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.115075    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf" event={"ID":"713a4870-e144-4960-bbff-0c4e623ba57c","Type":"ContainerDied","Data":"f5d7cb0236739c8c9a65c274dd95b566e5575c8e1517267b71b7bd6849edb152"}
Oct 31 04:15:21.122089 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.122071    4900 scope.go:122] "RemoveContainer" containerID="4222337db3f10d4ce86b971c8d7c0fff9e9e494b2f3906ca749e5e0b059f94f5"
Oct 31 04:15:21.157827 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.157607    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a9e5b0-9d34-468d-a70a-304974436a9b" path="/var/lib/kubelet/pods/64a9e5b0-9d34-468d-a70a-304974436a9b/volumes"
Oct 31 04:15:21.158194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.158171    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8797d751-7c89-4eb8-aaf9-abf0c81efcca" path="/var/lib/kubelet/pods/8797d751-7c89-4eb8-aaf9-abf0c81efcca/volumes"
Oct 31 04:15:21.158662 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.158640    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b45f245-d9e1-4476-bdb9-4a840c078444" path="/var/lib/kubelet/pods/8b45f245-d9e1-4476-bdb9-4a840c078444/volumes"
Oct 31 04:15:21.284058 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.284025    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"]
Oct 31 04:15:21.288486 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.288428    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-a5f70c-29890335-r92hf"]
Oct 31 04:15:21.322778 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.322187    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"
Oct 31 04:15:21.324538 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.324515    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"
Oct 31 04:15:21.341608 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.341578    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecfcb4c4-007c-4acc-a20c-092b12cfc44b-kube-api-access-nbczp\" (UniqueName: \"kubernetes.io/projected/ecfcb4c4-007c-4acc-a20c-092b12cfc44b-kube-api-access-nbczp\") pod \"ecfcb4c4-007c-4acc-a20c-092b12cfc44b\" (UID: \"ecfcb4c4-007c-4acc-a20c-092b12cfc44b\") "
Oct 31 04:15:21.341737 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.341633    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2146db6-ed0b-4776-930f-e1f915526586-kube-api-access-246b9\" (UniqueName: \"kubernetes.io/projected/a2146db6-ed0b-4776-930f-e1f915526586-kube-api-access-246b9\") pod \"a2146db6-ed0b-4776-930f-e1f915526586\" (UID: \"a2146db6-ed0b-4776-930f-e1f915526586\") "
Oct 31 04:15:21.341737 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.341682    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2146db6-ed0b-4776-930f-e1f915526586-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2146db6-ed0b-4776-930f-e1f915526586-maas-api-service-ca\") pod \"a2146db6-ed0b-4776-930f-e1f915526586\" (UID: \"a2146db6-ed0b-4776-930f-e1f915526586\") "
Oct 31 04:15:21.341865 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.341779    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecfcb4c4-007c-4acc-a20c-092b12cfc44b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfcb4c4-007c-4acc-a20c-092b12cfc44b-maas-api-service-ca\") pod \"ecfcb4c4-007c-4acc-a20c-092b12cfc44b\" (UID: \"ecfcb4c4-007c-4acc-a20c-092b12cfc44b\") "
Oct 31 04:15:21.342325 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.342299    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2146db6-ed0b-4776-930f-e1f915526586-maas-api-service-ca" pod "a2146db6-ed0b-4776-930f-e1f915526586" (UID: "a2146db6-ed0b-4776-930f-e1f915526586"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:21.342449 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.342429    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecfcb4c4-007c-4acc-a20c-092b12cfc44b-maas-api-service-ca" pod "ecfcb4c4-007c-4acc-a20c-092b12cfc44b" (UID: "ecfcb4c4-007c-4acc-a20c-092b12cfc44b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:21.342536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.342513    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2146db6-ed0b-4776-930f-e1f915526586-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:21.348819 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.344493    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2146db6-ed0b-4776-930f-e1f915526586-kube-api-access-246b9" pod "a2146db6-ed0b-4776-930f-e1f915526586" (UID: "a2146db6-ed0b-4776-930f-e1f915526586"). InnerVolumeSpecName "kube-api-access-246b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:21.348819 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.344714    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfcb4c4-007c-4acc-a20c-092b12cfc44b-kube-api-access-nbczp" pod "ecfcb4c4-007c-4acc-a20c-092b12cfc44b" (UID: "ecfcb4c4-007c-4acc-a20c-092b12cfc44b"). InnerVolumeSpecName "kube-api-access-nbczp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:21.443606 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.443574    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbczp\" (UniqueName: \"kubernetes.io/projected/ecfcb4c4-007c-4acc-a20c-092b12cfc44b-kube-api-access-nbczp\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:21.443606 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.443601    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-246b9\" (UniqueName: \"kubernetes.io/projected/a2146db6-ed0b-4776-930f-e1f915526586-kube-api-access-246b9\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:21.443606 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.443613    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfcb4c4-007c-4acc-a20c-092b12cfc44b-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:21.502680 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.502642    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"]
Oct 31 04:15:21.505561 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:21.505535    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-243a29-29890335-nwggm"]
Oct 31 04:15:22.124459 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:22.124428    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"
Oct 31 04:15:22.124940 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:22.124454    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r" event={"ID":"a2146db6-ed0b-4776-930f-e1f915526586","Type":"ContainerDied","Data":"94c0135df4a746d2e7daf5a086458c8283d3f5695edd3f314c70fefb6688d847"}
Oct 31 04:15:22.124940 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:22.124547    4900 scope.go:122] "RemoveContainer" containerID="548af04c1963798c2c0ea1f44b6fe48b229f656f8e19f55281698044bef18a3a"
Oct 31 04:15:22.126141 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:22.126119    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"
Oct 31 04:15:22.126271 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:22.126148    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw" event={"ID":"ecfcb4c4-007c-4acc-a20c-092b12cfc44b","Type":"ContainerDied","Data":"25465c427dd2a77642169114bdd26e6a02997e86dd2547748c5459319e6c49af"}
Oct 31 04:15:22.134943 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:22.134925    4900 scope.go:122] "RemoveContainer" containerID="b01d9ab50569dd2e8dfae5188f6ff0225c67e5a1ee67ff2c06e6b625ccaf810a"
Oct 31 04:15:22.154892 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:22.154870    4900 patch_prober.go:29] interesting pod/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:15:22.155027 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:22.154912    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px" podUID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:15:22.288526 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:22.288495    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"]
Oct 31 04:15:22.291171 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:22.291150    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-93f411-29890335-xhmfw"]
Oct 31 04:15:22.400833 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:22.400743    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"]
Oct 31 04:15:22.403440 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:22.403421    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7778d0-29890335-v999r"]
Oct 31 04:15:23.017519 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:23.017493    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:15:23.139485 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:23.139456    4900 generic.go:356] "Generic (PLEG): container finished" podID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerID="e8419a3d511e712c305033d9a5f013a54b9a48c9d956ede40006ff61fcaf2fce" exitCode=0
Oct 31 04:15:23.139891 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:23.139533    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px" event={"ID":"e6b94bdc-4427-400c-ad3f-5032aa82a83f","Type":"ContainerDied","Data":"e8419a3d511e712c305033d9a5f013a54b9a48c9d956ede40006ff61fcaf2fce"}
Oct 31 04:15:23.139992 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:23.139965    4900 scope.go:122] "RemoveContainer" containerID="e8419a3d511e712c305033d9a5f013a54b9a48c9d956ede40006ff61fcaf2fce"
Oct 31 04:15:23.141709 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:23.141678    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:15:23.141836 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:23.141748    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:15:23.141836 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:23.141808    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:15:23.156782 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:23.156756    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="713a4870-e144-4960-bbff-0c4e623ba57c" path="/var/lib/kubelet/pods/713a4870-e144-4960-bbff-0c4e623ba57c/volumes"
Oct 31 04:15:23.157564 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:23.157541    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2146db6-ed0b-4776-930f-e1f915526586" path="/var/lib/kubelet/pods/a2146db6-ed0b-4776-930f-e1f915526586/volumes"
Oct 31 04:15:23.159823 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:23.158309    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4447ca2-888d-4950-b75d-b6149d71791a" path="/var/lib/kubelet/pods/e4447ca2-888d-4950-b75d-b6149d71791a/volumes"
Oct 31 04:15:23.159823 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:23.158926    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b" path="/var/lib/kubelet/pods/ecfcb4c4-007c-4acc-a20c-092b12cfc44b/volumes"
Oct 31 04:15:24.180456 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:24.180409    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px" event={"ID":"e6b94bdc-4427-400c-ad3f-5032aa82a83f","Type":"ContainerStarted","Data":"de856a78d19581d7d2d9360123956461bb7ea74589041e60fb8a60c364ce0b14"}
Oct 31 04:15:24.711004 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:24.710970    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:24.725290 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:24.725259    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:15:24.725495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:24.725429    4900 patch_prober.go:29] interesting pod/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 04:15:24.725495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:24.725483    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px" podUID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 04:15:25.724917 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:25.724878    4900 patch_prober.go:29] interesting pod/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 04:15:25.725340 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:25.724945    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px" podUID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 04:15:26.724735 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:26.724700    4900 patch_prober.go:29] interesting pod/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 04:15:26.724950 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:26.724761    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px" podUID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 04:15:27.725006 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:27.724970    4900 patch_prober.go:29] interesting pod/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 04:15:27.725413 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:27.725035    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px" podUID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 04:15:28.724428 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:28.724392    4900 patch_prober.go:29] interesting pod/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 04:15:28.724583 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:28.724453    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px" podUID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 04:15:28.860030 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:28.859998    4900 scope.go:122] "RemoveContainer" containerID="71df5fb271c76804349383bc2c54c1edefa19834b9c85e47d555af66ea0802df"
Oct 31 04:15:29.582338 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:29.582289    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:29.728738 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:29.728705    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:15:29.729014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:29.728996    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:15:29.729722 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:29.729707    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:15:33.635454 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:33.635416    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"]
Oct 31 04:15:33.635906 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:33.635621    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn" podUID="9ce42511-8082-4e74-9822-85d39f2536f3" containerName="maas-api" containerID="cri-o://340638882f81b1dcd0aae10efc60c19e8a512cb2ec7d0e547daf33752d1a1563" gracePeriod=30
Oct 31 04:15:33.838507 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:33.838477    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:33.920459 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:33.920432    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:15:33.944589 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:33.944559    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-maas-api-tls\") pod \"9ce42511-8082-4e74-9822-85d39f2536f3\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") "
Oct 31 04:15:33.944778 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:33.944655    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ce42511-8082-4e74-9822-85d39f2536f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce42511-8082-4e74-9822-85d39f2536f3-openshift-service-ca\") pod \"9ce42511-8082-4e74-9822-85d39f2536f3\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") "
Oct 31 04:15:33.944778 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:33.944743    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ce42511-8082-4e74-9822-85d39f2536f3-kube-api-access-p822s\" (UniqueName: \"kubernetes.io/projected/9ce42511-8082-4e74-9822-85d39f2536f3-kube-api-access-p822s\") pod \"9ce42511-8082-4e74-9822-85d39f2536f3\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") "
Oct 31 04:15:33.944952 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:33.944823    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-metrics-certs\") pod \"9ce42511-8082-4e74-9822-85d39f2536f3\" (UID: \"9ce42511-8082-4e74-9822-85d39f2536f3\") "
Oct 31 04:15:33.945100 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:33.945067    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ce42511-8082-4e74-9822-85d39f2536f3-openshift-service-ca" pod "9ce42511-8082-4e74-9822-85d39f2536f3" (UID: "9ce42511-8082-4e74-9822-85d39f2536f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:33.945436 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:33.945419    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ce42511-8082-4e74-9822-85d39f2536f3-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:33.946706 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:33.946682    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-maas-api-tls" pod "9ce42511-8082-4e74-9822-85d39f2536f3" (UID: "9ce42511-8082-4e74-9822-85d39f2536f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:15:33.946920 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:33.946899    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ce42511-8082-4e74-9822-85d39f2536f3-kube-api-access-p822s" pod "9ce42511-8082-4e74-9822-85d39f2536f3" (UID: "9ce42511-8082-4e74-9822-85d39f2536f3"). InnerVolumeSpecName "kube-api-access-p822s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:33.946979 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:33.946915    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-metrics-certs" pod "9ce42511-8082-4e74-9822-85d39f2536f3" (UID: "9ce42511-8082-4e74-9822-85d39f2536f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:15:34.047819 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:34.046428    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:34.047819 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:34.046470    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p822s\" (UniqueName: \"kubernetes.io/projected/9ce42511-8082-4e74-9822-85d39f2536f3-kube-api-access-p822s\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:34.047819 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:34.046485    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ce42511-8082-4e74-9822-85d39f2536f3-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:34.245951 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:34.245914    4900 generic.go:356] "Generic (PLEG): container finished" podID="9ce42511-8082-4e74-9822-85d39f2536f3" containerID="340638882f81b1dcd0aae10efc60c19e8a512cb2ec7d0e547daf33752d1a1563" exitCode=0
Oct 31 04:15:34.246164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:34.245964    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"
Oct 31 04:15:34.246164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:34.246005    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn" event={"ID":"9ce42511-8082-4e74-9822-85d39f2536f3","Type":"ContainerDied","Data":"340638882f81b1dcd0aae10efc60c19e8a512cb2ec7d0e547daf33752d1a1563"}
Oct 31 04:15:34.246164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:34.246067    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn" event={"ID":"9ce42511-8082-4e74-9822-85d39f2536f3","Type":"ContainerDied","Data":"1f3203671d05e944f95f3d2e19e3d390fc0ae05c2b1f2a862b6db92a041dba0a"}
Oct 31 04:15:34.246164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:34.246127    4900 scope.go:122] "RemoveContainer" containerID="340638882f81b1dcd0aae10efc60c19e8a512cb2ec7d0e547daf33752d1a1563"
Oct 31 04:15:34.256293 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:34.256277    4900 scope.go:122] "RemoveContainer" containerID="340638882f81b1dcd0aae10efc60c19e8a512cb2ec7d0e547daf33752d1a1563"
Oct 31 04:15:34.256647 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:34.256621    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340638882f81b1dcd0aae10efc60c19e8a512cb2ec7d0e547daf33752d1a1563\": container with ID starting with 340638882f81b1dcd0aae10efc60c19e8a512cb2ec7d0e547daf33752d1a1563 not found: ID does not exist" containerID="340638882f81b1dcd0aae10efc60c19e8a512cb2ec7d0e547daf33752d1a1563"
Oct 31 04:15:34.256759 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:34.256659    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340638882f81b1dcd0aae10efc60c19e8a512cb2ec7d0e547daf33752d1a1563"} err="failed to get container status \"340638882f81b1dcd0aae10efc60c19e8a512cb2ec7d0e547daf33752d1a1563\": rpc error: code = NotFound desc = could not find container \"340638882f81b1dcd0aae10efc60c19e8a512cb2ec7d0e547daf33752d1a1563\": container with ID starting with 340638882f81b1dcd0aae10efc60c19e8a512cb2ec7d0e547daf33752d1a1563 not found: ID does not exist"
Oct 31 04:15:34.413620 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:34.413574    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"]
Oct 31 04:15:34.463539 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:34.463508    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d247af-6b8cbd4c98-dhbzn"]
Oct 31 04:15:35.153877 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:35.153846    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ce42511-8082-4e74-9822-85d39f2536f3" path="/var/lib/kubelet/pods/9ce42511-8082-4e74-9822-85d39f2536f3/volumes"
Oct 31 04:15:35.724761 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:35.724725    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:35.942556 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:35.942526    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:36.335779 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:36.335744    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:36.984611 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:36.984577    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:15:38.721003 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:38.720967    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:40.376451 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.376419    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:40.386533 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.386507    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"]
Oct 31 04:15:40.387194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387178    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b" containerName="cleanup"
Oct 31 04:15:40.387253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387195    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b" containerName="cleanup"
Oct 31 04:15:40.387253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387208    4900 cpu_manager.go:414] "containerMap: removing container" podUID="713a4870-e144-4960-bbff-0c4e623ba57c" containerName="cleanup"
Oct 31 04:15:40.387253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387214    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713a4870-e144-4960-bbff-0c4e623ba57c" containerName="cleanup"
Oct 31 04:15:40.387253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387223    4900 cpu_manager.go:414] "containerMap: removing container" podUID="a2146db6-ed0b-4776-930f-e1f915526586" containerName="cleanup"
Oct 31 04:15:40.387253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387228    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2146db6-ed0b-4776-930f-e1f915526586" containerName="cleanup"
Oct 31 04:15:40.387253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387233    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b" containerName="cleanup"
Oct 31 04:15:40.387253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387238    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b" containerName="cleanup"
Oct 31 04:15:40.387253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387248    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b" containerName="cleanup"
Oct 31 04:15:40.387253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387253    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387266    4900 cpu_manager.go:414] "containerMap: removing container" podUID="e4447ca2-888d-4950-b75d-b6149d71791a" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387271    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4447ca2-888d-4950-b75d-b6149d71791a" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387280    4900 cpu_manager.go:414] "containerMap: removing container" podUID="64a9e5b0-9d34-468d-a70a-304974436a9b" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387285    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a9e5b0-9d34-468d-a70a-304974436a9b" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387292    4900 cpu_manager.go:414] "containerMap: removing container" podUID="64a9e5b0-9d34-468d-a70a-304974436a9b" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387296    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a9e5b0-9d34-468d-a70a-304974436a9b" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387304    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8b45f245-d9e1-4476-bdb9-4a840c078444" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387308    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b45f245-d9e1-4476-bdb9-4a840c078444" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387315    4900 cpu_manager.go:414] "containerMap: removing container" podUID="713a4870-e144-4960-bbff-0c4e623ba57c" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387319    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713a4870-e144-4960-bbff-0c4e623ba57c" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387327    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8b45f245-d9e1-4476-bdb9-4a840c078444" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387331    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b45f245-d9e1-4476-bdb9-4a840c078444" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387339    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8797d751-7c89-4eb8-aaf9-abf0c81efcca" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387343    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8797d751-7c89-4eb8-aaf9-abf0c81efcca" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387350    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8b45f245-d9e1-4476-bdb9-4a840c078444" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387353    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b45f245-d9e1-4476-bdb9-4a840c078444" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387361    4900 cpu_manager.go:414] "containerMap: removing container" podUID="e4447ca2-888d-4950-b75d-b6149d71791a" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387365    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4447ca2-888d-4950-b75d-b6149d71791a" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387373    4900 cpu_manager.go:414] "containerMap: removing container" podUID="64a9e5b0-9d34-468d-a70a-304974436a9b" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387377    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a9e5b0-9d34-468d-a70a-304974436a9b" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387391    4900 cpu_manager.go:414] "containerMap: removing container" podUID="a2146db6-ed0b-4776-930f-e1f915526586" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387395    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2146db6-ed0b-4776-930f-e1f915526586" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387410    4900 cpu_manager.go:414] "containerMap: removing container" podUID="9ce42511-8082-4e74-9822-85d39f2536f3" containerName="maas-api"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387414    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ce42511-8082-4e74-9822-85d39f2536f3" containerName="maas-api"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387421    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8797d751-7c89-4eb8-aaf9-abf0c81efcca" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387426    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8797d751-7c89-4eb8-aaf9-abf0c81efcca" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387506    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b45f245-d9e1-4476-bdb9-4a840c078444" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387514    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8797d751-7c89-4eb8-aaf9-abf0c81efcca" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387519    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4447ca2-888d-4950-b75d-b6149d71791a" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387525    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b45f245-d9e1-4476-bdb9-4a840c078444" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387530    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4447ca2-888d-4950-b75d-b6149d71791a" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387537    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a9e5b0-9d34-468d-a70a-304974436a9b" containerName="cleanup"
Oct 31 04:15:40.387532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387543    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2146db6-ed0b-4776-930f-e1f915526586" containerName="cleanup"
Oct 31 04:15:40.388607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387551    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ce42511-8082-4e74-9822-85d39f2536f3" containerName="maas-api"
Oct 31 04:15:40.388607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387557    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8797d751-7c89-4eb8-aaf9-abf0c81efcca" containerName="cleanup"
Oct 31 04:15:40.388607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387562    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b" containerName="cleanup"
Oct 31 04:15:40.388607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387566    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="713a4870-e144-4960-bbff-0c4e623ba57c" containerName="cleanup"
Oct 31 04:15:40.388607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387571    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2146db6-ed0b-4776-930f-e1f915526586" containerName="cleanup"
Oct 31 04:15:40.388607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387575    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b" containerName="cleanup"
Oct 31 04:15:40.388607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387582    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2146db6-ed0b-4776-930f-e1f915526586" containerName="cleanup"
Oct 31 04:15:40.388607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387588    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="713a4870-e144-4960-bbff-0c4e623ba57c" containerName="cleanup"
Oct 31 04:15:40.388607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.387593    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a9e5b0-9d34-468d-a70a-304974436a9b" containerName="cleanup"
Oct 31 04:15:40.390249 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.390233    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.442772 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.442722    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c42c43-openshift-default-dockercfg-qzh7g\""
Oct 31 04:15:40.461921 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.461889    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-envoy\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.462092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.461947    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-data\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.462092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.462062    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-workload-certs\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.462179 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.462162    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-workload-socket\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.462219 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.462183    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zxwj\" (UniqueName: \"kubernetes.io/projected/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-kube-api-access-4zxwj\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.462261 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.462249    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-credential-socket\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.462309 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.462294    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istiod-ca-cert\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.462439 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.462426    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-token\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.462511 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.462488    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-podinfo\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.495988 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.495954    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"]
Oct 31 04:15:40.563451 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.563418    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-workload-socket\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.563451 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.563458    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zxwj\" (UniqueName: \"kubernetes.io/projected/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-kube-api-access-4zxwj\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.563897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.563505    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-credential-socket\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.563897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.563546    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istiod-ca-cert\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.563897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.563634    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-token\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.563897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.563721    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-podinfo\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.563897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.563761    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-envoy\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.563897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.563852    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-data\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.563897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.563875    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-workload-socket\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.564276 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.563904    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-workload-certs\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.564276 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.563989    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-credential-socket\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.564276 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.564183    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-workload-certs\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.564401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.564296    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-data\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.564450 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.564420    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istiod-ca-cert\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.565984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.565965    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-envoy\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.566311 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.566290    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-podinfo\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.589546 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.589523    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:40.621272 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.621242    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zxwj\" (UniqueName: \"kubernetes.io/projected/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-kube-api-access-4zxwj\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.621888 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.621860    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-token\") pod \"e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") " pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.703215 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.703130    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:40.805432 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:40.805402    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:41.087263 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:41.087143    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:41.119320 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:15:41.119291    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed827b7_7ac6_412d_ade0_70ce0d8cc399.slice/crio-fc9a3b85def1a48dca185c61518da396b0072a25df5468dc65efb5d9a2f5dc33 WatchSource:0}: Error finding container fc9a3b85def1a48dca185c61518da396b0072a25df5468dc65efb5d9a2f5dc33: Status 404 returned error can't find the container with id fc9a3b85def1a48dca185c61518da396b0072a25df5468dc65efb5d9a2f5dc33
Oct 31 04:15:41.121490 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:41.121465    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"]
Oct 31 04:15:41.122145 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:41.122114    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:15:41.122215 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:41.122180    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:15:41.122215 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:41.122206    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:15:41.293361 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:41.293317    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r" event={"ID":"6ed827b7-7ac6-412d-ade0-70ce0d8cc399","Type":"ContainerStarted","Data":"a096ad3d8e428037aaf1a67313426834d9b7411c2a2119c8db836703ff6f879b"}
Oct 31 04:15:41.293533 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:41.293385    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r" event={"ID":"6ed827b7-7ac6-412d-ade0-70ce0d8cc399","Type":"ContainerStarted","Data":"fc9a3b85def1a48dca185c61518da396b0072a25df5468dc65efb5d9a2f5dc33"}
Oct 31 04:15:41.409237 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:41.409114    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r" podStartSLOduration=1.409097182 podStartE2EDuration="1.409097182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:41.408822772 +0000 UTC m=+4094.375156115" watchObservedRunningTime="2026-10-31 04:15:41.409097182 +0000 UTC m=+4094.375430527"
Oct 31 04:15:41.703941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:41.703840    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:42.710089 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:42.710059    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:43.306231 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:43.306202    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:43.307292 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:43.307267    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:15:43.724897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:43.724817    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:44.289549 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:44.289515    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:44.510663 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:44.510621    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:46.385094 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:46.385049    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:47.546782 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:47.546748    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:47.753139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:47.753097    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:49.755081 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:49.755047    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:51.034803 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:51.034750    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"]
Oct 31 04:15:51.145415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:51.145382    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"]
Oct 31 04:15:51.490860 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:51.490779    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:51.703242 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:51.703209    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:52.126902 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:52.126867    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:55.182106 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:55.182067    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:56.055065 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.054855    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv" podUID="f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473" containerName="payload-processing" containerID="cri-o://932d92e8c1a33cd78f1d5889eac91ac1b7474f119543f1ad2ec426594f8f990f" gracePeriod=25
Oct 31 04:15:56.153767 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.153101    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5" podUID="695cf523-b73f-4e3d-999e-eb0b36ad3ed5" containerName="payload-pre-processing" containerID="cri-o://9e82864daa0f5e196936d3f0c630ba1f2c4de2a54b8af1a855dcc455ced3e15e" gracePeriod=25
Oct 31 04:15:56.376450 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.376183    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"
Oct 31 04:15:56.396871 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.396743    4900 generic.go:356] "Generic (PLEG): container finished" podID="695cf523-b73f-4e3d-999e-eb0b36ad3ed5" containerID="9e82864daa0f5e196936d3f0c630ba1f2c4de2a54b8af1a855dcc455ced3e15e" exitCode=0
Oct 31 04:15:56.397048 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.396899    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5" event={"ID":"695cf523-b73f-4e3d-999e-eb0b36ad3ed5","Type":"ContainerDied","Data":"9e82864daa0f5e196936d3f0c630ba1f2c4de2a54b8af1a855dcc455ced3e15e"}
Oct 31 04:15:56.407079 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.407051    4900 generic.go:356] "Generic (PLEG): container finished" podID="f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473" containerID="932d92e8c1a33cd78f1d5889eac91ac1b7474f119543f1ad2ec426594f8f990f" exitCode=0
Oct 31 04:15:56.407237 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.407094    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv" event={"ID":"f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473","Type":"ContainerDied","Data":"932d92e8c1a33cd78f1d5889eac91ac1b7474f119543f1ad2ec426594f8f990f"}
Oct 31 04:15:56.407237 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.407139    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv" event={"ID":"f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473","Type":"ContainerDied","Data":"bdad99e1d997f4936f28f02e57c569cabb08572a2356e926500a830abc92e4a8"}
Oct 31 04:15:56.407237 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.407169    4900 scope.go:122] "RemoveContainer" containerID="932d92e8c1a33cd78f1d5889eac91ac1b7474f119543f1ad2ec426594f8f990f"
Oct 31 04:15:56.407424 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.407325    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"
Oct 31 04:15:56.417597 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.417428    4900 scope.go:122] "RemoveContainer" containerID="932d92e8c1a33cd78f1d5889eac91ac1b7474f119543f1ad2ec426594f8f990f"
Oct 31 04:15:56.417834 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:15:56.417753    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932d92e8c1a33cd78f1d5889eac91ac1b7474f119543f1ad2ec426594f8f990f\": container with ID starting with 932d92e8c1a33cd78f1d5889eac91ac1b7474f119543f1ad2ec426594f8f990f not found: ID does not exist" containerID="932d92e8c1a33cd78f1d5889eac91ac1b7474f119543f1ad2ec426594f8f990f"
Oct 31 04:15:56.417834 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.417780    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932d92e8c1a33cd78f1d5889eac91ac1b7474f119543f1ad2ec426594f8f990f"} err="failed to get container status \"932d92e8c1a33cd78f1d5889eac91ac1b7474f119543f1ad2ec426594f8f990f\": rpc error: code = NotFound desc = could not find container \"932d92e8c1a33cd78f1d5889eac91ac1b7474f119543f1ad2ec426594f8f990f\": container with ID starting with 932d92e8c1a33cd78f1d5889eac91ac1b7474f119543f1ad2ec426594f8f990f not found: ID does not exist"
Oct 31 04:15:56.473053 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.473025    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"
Oct 31 04:15:56.507316 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.507289    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473-kube-api-access-t6nsv\" (UniqueName: \"kubernetes.io/projected/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473-kube-api-access-t6nsv\") pod \"f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473\" (UID: \"f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473\") "
Oct 31 04:15:56.507470 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.507399    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473-plugins-config-volume\") pod \"f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473\" (UID: \"f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473\") "
Oct 31 04:15:56.507884 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.507846    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473-plugins-config-volume" pod "f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473" (UID: "f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:56.508135 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.508113    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:56.509371 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.509349    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473-kube-api-access-t6nsv" pod "f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473" (UID: "f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473"). InnerVolumeSpecName "kube-api-access-t6nsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:56.609360 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.609272    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695cf523-b73f-4e3d-999e-eb0b36ad3ed5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695cf523-b73f-4e3d-999e-eb0b36ad3ed5-plugins-config-volume\") pod \"695cf523-b73f-4e3d-999e-eb0b36ad3ed5\" (UID: \"695cf523-b73f-4e3d-999e-eb0b36ad3ed5\") "
Oct 31 04:15:56.609360 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.609322    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695cf523-b73f-4e3d-999e-eb0b36ad3ed5-kube-api-access-7r5zf\" (UniqueName: \"kubernetes.io/projected/695cf523-b73f-4e3d-999e-eb0b36ad3ed5-kube-api-access-7r5zf\") pod \"695cf523-b73f-4e3d-999e-eb0b36ad3ed5\" (UID: \"695cf523-b73f-4e3d-999e-eb0b36ad3ed5\") "
Oct 31 04:15:56.609678 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.609653    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695cf523-b73f-4e3d-999e-eb0b36ad3ed5-plugins-config-volume" pod "695cf523-b73f-4e3d-999e-eb0b36ad3ed5" (UID: "695cf523-b73f-4e3d-999e-eb0b36ad3ed5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:56.609897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.609870    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/695cf523-b73f-4e3d-999e-eb0b36ad3ed5-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:56.609897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.609894    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6nsv\" (UniqueName: \"kubernetes.io/projected/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473-kube-api-access-t6nsv\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:56.611401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.611380    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695cf523-b73f-4e3d-999e-eb0b36ad3ed5-kube-api-access-7r5zf" pod "695cf523-b73f-4e3d-999e-eb0b36ad3ed5" (UID: "695cf523-b73f-4e3d-999e-eb0b36ad3ed5"). InnerVolumeSpecName "kube-api-access-7r5zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:56.716815 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.711901    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r5zf\" (UniqueName: \"kubernetes.io/projected/695cf523-b73f-4e3d-999e-eb0b36ad3ed5-kube-api-access-7r5zf\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:56.902294 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.902207    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"]
Oct 31 04:15:56.905619 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:56.905595    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f0e007-68bb6c7d5c-729bv"]
Oct 31 04:15:57.169988 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:57.169906    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473" path="/var/lib/kubelet/pods/f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473/volumes"
Oct 31 04:15:57.415027 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:57.414996    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"
Oct 31 04:15:57.415469 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:57.415037    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5" event={"ID":"695cf523-b73f-4e3d-999e-eb0b36ad3ed5","Type":"ContainerDied","Data":"b33750bbe595a1f3f07f8ff0065d3bfd494e2014b4b7736f64bb8c361474c81d"}
Oct 31 04:15:57.415469 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:57.415098    4900 scope.go:122] "RemoveContainer" containerID="9e82864daa0f5e196936d3f0c630ba1f2c4de2a54b8af1a855dcc455ced3e15e"
Oct 31 04:15:57.579464 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:57.579429    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"]
Oct 31 04:15:57.584463 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:57.584440    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f0e007-ffdb68675-5dpw5"]
Oct 31 04:15:57.618092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:57.618050    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:15:59.155748 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:15:59.155711    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695cf523-b73f-4e3d-999e-eb0b36ad3ed5" path="/var/lib/kubelet/pods/695cf523-b73f-4e3d-999e-eb0b36ad3ed5/volumes"
Oct 31 04:16:01.006040 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:01.006001    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"]
Oct 31 04:16:01.006520 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:01.006238    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl" podUID="86e324aa-c2e3-4524-a3f0-2d2af88b6063" containerName="istio-proxy" containerID="cri-o://8dc4727f150855ac72e551b486df4d8f6af8e0d3965eccbdf34c2bca42ed3275" gracePeriod=30
Oct 31 04:16:01.056263 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:01.056232    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"]
Oct 31 04:16:01.056474 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:01.056428    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9" podUID="191f5b84-f796-4f2b-8cba-a8948506b26b" containerName="istio-proxy" containerID="cri-o://e80111bd79a1f16a5709f13fa5fbfb8c7df78e325cffe59edc5682cc526e81fc" gracePeriod=30
Oct 31 04:16:01.443588 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:01.443504    4900 generic.go:356] "Generic (PLEG): container finished" podID="c0e37b79-c673-4bb9-a740-eda25789449a" containerID="569f4ec56e74a1e590dc0b2b2981dccbaf482b9ecb0d49aff3ea33e330735a10" exitCode=0
Oct 31 04:16:01.443588 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:01.443578    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq" event={"ID":"c0e37b79-c673-4bb9-a740-eda25789449a","Type":"ContainerDied","Data":"569f4ec56e74a1e590dc0b2b2981dccbaf482b9ecb0d49aff3ea33e330735a10"}
Oct 31 04:16:01.444014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:01.443995    4900 scope.go:122] "RemoveContainer" containerID="569f4ec56e74a1e590dc0b2b2981dccbaf482b9ecb0d49aff3ea33e330735a10"
Oct 31 04:16:01.445517 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:01.445483    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:16:01.445629 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:01.445552    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:16:01.445629 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:01.445587    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:16:02.451709 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:02.451671    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq" event={"ID":"c0e37b79-c673-4bb9-a740-eda25789449a","Type":"ContainerStarted","Data":"a72b1b27134fe96ac1bf6789a7cee605154a8cb6b4e925017e52938ebc70baa5"}
Oct 31 04:16:03.021422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:03.021393    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:03.160524 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:03.160488    4900 patch_prober.go:29] interesting pod/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 04:16:03.160837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:03.160810    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq" podUID="c0e37b79-c673-4bb9-a740-eda25789449a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 04:16:03.167017 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:03.166991    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:16:04.159430 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:04.159395    4900 patch_prober.go:29] interesting pod/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 04:16:04.160097 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:04.160059    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq" podUID="c0e37b79-c673-4bb9-a740-eda25789449a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 04:16:05.159554 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:05.159513    4900 patch_prober.go:29] interesting pod/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 04:16:05.160124 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:05.159578    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq" podUID="c0e37b79-c673-4bb9-a740-eda25789449a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 04:16:05.659664 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:05.659625    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:06.159741 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.159314    4900 patch_prober.go:29] interesting pod/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 04:16:06.159741 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.159383    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq" podUID="c0e37b79-c673-4bb9-a740-eda25789449a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 04:16:06.372643 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.372614    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:16:06.408190 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.408167    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:16:06.470894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.470863    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-podinfo\") pod \"191f5b84-f796-4f2b-8cba-a8948506b26b\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") "
Oct 31 04:16:06.471081 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.470921    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-credential-socket\") pod \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") "
Oct 31 04:16:06.471081 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.470991    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-workload-socket\") pod \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") "
Oct 31 04:16:06.471081 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471024    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-workload-socket\") pod \"191f5b84-f796-4f2b-8cba-a8948506b26b\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") "
Oct 31 04:16:06.471264 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471112    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/191f5b84-f796-4f2b-8cba-a8948506b26b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191f5b84-f796-4f2b-8cba-a8948506b26b-istiod-ca-cert\") pod \"191f5b84-f796-4f2b-8cba-a8948506b26b\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") "
Oct 31 04:16:06.471264 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471149    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-envoy\") pod \"191f5b84-f796-4f2b-8cba-a8948506b26b\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") "
Oct 31 04:16:06.471264 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471185    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istiod-ca-cert\") pod \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") "
Oct 31 04:16:06.471264 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471224    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-token\" (UniqueName: \"kubernetes.io/projected/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-token\") pod \"191f5b84-f796-4f2b-8cba-a8948506b26b\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") "
Oct 31 04:16:06.471482 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471282    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-data\") pod \"191f5b84-f796-4f2b-8cba-a8948506b26b\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") "
Oct 31 04:16:06.471482 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471333    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191f5b84-f796-4f2b-8cba-a8948506b26b-kube-api-access-mjj7v\" (UniqueName: \"kubernetes.io/projected/191f5b84-f796-4f2b-8cba-a8948506b26b-kube-api-access-mjj7v\") pod \"191f5b84-f796-4f2b-8cba-a8948506b26b\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") "
Oct 31 04:16:06.471482 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471403    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-workload-certs\") pod \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") "
Oct 31 04:16:06.471482 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471433    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e324aa-c2e3-4524-a3f0-2d2af88b6063-kube-api-access-ghrvx\" (UniqueName: \"kubernetes.io/projected/86e324aa-c2e3-4524-a3f0-2d2af88b6063-kube-api-access-ghrvx\") pod \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") "
Oct 31 04:16:06.471698 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471507    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-token\" (UniqueName: \"kubernetes.io/projected/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-token\") pod \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") "
Oct 31 04:16:06.471698 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471571    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-credential-socket\") pod \"191f5b84-f796-4f2b-8cba-a8948506b26b\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") "
Oct 31 04:16:06.471698 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471601    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-data\") pod \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") "
Oct 31 04:16:06.471698 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471661    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-envoy\") pod \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") "
Oct 31 04:16:06.471698 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471695    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-podinfo\") pod \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\" (UID: \"86e324aa-c2e3-4524-a3f0-2d2af88b6063\") "
Oct 31 04:16:06.471985 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.471748    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-workload-certs\") pod \"191f5b84-f796-4f2b-8cba-a8948506b26b\" (UID: \"191f5b84-f796-4f2b-8cba-a8948506b26b\") "
Oct 31 04:16:06.472372 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.472305    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-data" pod "191f5b84-f796-4f2b-8cba-a8948506b26b" (UID: "191f5b84-f796-4f2b-8cba-a8948506b26b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:06.472372 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.472318    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-workload-socket" pod "191f5b84-f796-4f2b-8cba-a8948506b26b" (UID: "191f5b84-f796-4f2b-8cba-a8948506b26b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:06.472585 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.472465    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.472585 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.472514    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.472732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.472649    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-workload-socket" pod "86e324aa-c2e3-4524-a3f0-2d2af88b6063" (UID: "86e324aa-c2e3-4524-a3f0-2d2af88b6063"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:06.472732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.472692    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-credential-socket" pod "86e324aa-c2e3-4524-a3f0-2d2af88b6063" (UID: "86e324aa-c2e3-4524-a3f0-2d2af88b6063"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:06.472866 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.472746    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-credential-socket" pod "191f5b84-f796-4f2b-8cba-a8948506b26b" (UID: "191f5b84-f796-4f2b-8cba-a8948506b26b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:06.472929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.472908    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istiod-ca-cert" pod "86e324aa-c2e3-4524-a3f0-2d2af88b6063" (UID: "86e324aa-c2e3-4524-a3f0-2d2af88b6063"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:16:06.473056 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.473032    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-workload-certs" pod "191f5b84-f796-4f2b-8cba-a8948506b26b" (UID: "191f5b84-f796-4f2b-8cba-a8948506b26b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:06.473640 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.473259    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191f5b84-f796-4f2b-8cba-a8948506b26b-istiod-ca-cert" pod "191f5b84-f796-4f2b-8cba-a8948506b26b" (UID: "191f5b84-f796-4f2b-8cba-a8948506b26b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:16:06.473640 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.473311    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-data" pod "86e324aa-c2e3-4524-a3f0-2d2af88b6063" (UID: "86e324aa-c2e3-4524-a3f0-2d2af88b6063"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:06.473640 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.473478    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-workload-certs" pod "86e324aa-c2e3-4524-a3f0-2d2af88b6063" (UID: "86e324aa-c2e3-4524-a3f0-2d2af88b6063"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:06.474850 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.474816    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-podinfo" pod "191f5b84-f796-4f2b-8cba-a8948506b26b" (UID: "191f5b84-f796-4f2b-8cba-a8948506b26b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:16:06.475057 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.475033    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191f5b84-f796-4f2b-8cba-a8948506b26b-kube-api-access-mjj7v" pod "191f5b84-f796-4f2b-8cba-a8948506b26b" (UID: "191f5b84-f796-4f2b-8cba-a8948506b26b"). InnerVolumeSpecName "kube-api-access-mjj7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:06.475703 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.475685    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-envoy" pod "191f5b84-f796-4f2b-8cba-a8948506b26b" (UID: "191f5b84-f796-4f2b-8cba-a8948506b26b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:06.475833 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.475811    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e324aa-c2e3-4524-a3f0-2d2af88b6063-kube-api-access-ghrvx" pod "86e324aa-c2e3-4524-a3f0-2d2af88b6063" (UID: "86e324aa-c2e3-4524-a3f0-2d2af88b6063"). InnerVolumeSpecName "kube-api-access-ghrvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:06.475908 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.475892    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-token" pod "86e324aa-c2e3-4524-a3f0-2d2af88b6063" (UID: "86e324aa-c2e3-4524-a3f0-2d2af88b6063"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:06.476134 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.476119    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-envoy" pod "86e324aa-c2e3-4524-a3f0-2d2af88b6063" (UID: "86e324aa-c2e3-4524-a3f0-2d2af88b6063"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:16:06.476198 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.476186    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-token" pod "191f5b84-f796-4f2b-8cba-a8948506b26b" (UID: "191f5b84-f796-4f2b-8cba-a8948506b26b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:06.476563 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.476506    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-podinfo" pod "86e324aa-c2e3-4524-a3f0-2d2af88b6063" (UID: "86e324aa-c2e3-4524-a3f0-2d2af88b6063"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:16:06.483984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.483834    4900 generic.go:356] "Generic (PLEG): container finished" podID="86e324aa-c2e3-4524-a3f0-2d2af88b6063" containerID="8dc4727f150855ac72e551b486df4d8f6af8e0d3965eccbdf34c2bca42ed3275" exitCode=0
Oct 31 04:16:06.483984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.483935    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"
Oct 31 04:16:06.483984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.483968    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl" event={"ID":"86e324aa-c2e3-4524-a3f0-2d2af88b6063","Type":"ContainerDied","Data":"8dc4727f150855ac72e551b486df4d8f6af8e0d3965eccbdf34c2bca42ed3275"}
Oct 31 04:16:06.484159 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.484015    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl" event={"ID":"86e324aa-c2e3-4524-a3f0-2d2af88b6063","Type":"ContainerDied","Data":"3c62ebb282f4feae0e14a95f92bb1c3eee5748d6dab24cf4a1aa1020a8d601d6"}
Oct 31 04:16:06.484159 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.484042    4900 scope.go:122] "RemoveContainer" containerID="8dc4727f150855ac72e551b486df4d8f6af8e0d3965eccbdf34c2bca42ed3275"
Oct 31 04:16:06.485895 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.485515    4900 generic.go:356] "Generic (PLEG): container finished" podID="191f5b84-f796-4f2b-8cba-a8948506b26b" containerID="e80111bd79a1f16a5709f13fa5fbfb8c7df78e325cffe59edc5682cc526e81fc" exitCode=0
Oct 31 04:16:06.485895 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.485553    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9" event={"ID":"191f5b84-f796-4f2b-8cba-a8948506b26b","Type":"ContainerDied","Data":"e80111bd79a1f16a5709f13fa5fbfb8c7df78e325cffe59edc5682cc526e81fc"}
Oct 31 04:16:06.485895 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.485581    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9" event={"ID":"191f5b84-f796-4f2b-8cba-a8948506b26b","Type":"ContainerDied","Data":"b892cc077d2e45aaae4d1feed4135d40066450dda979f46472f1a5c10952b85e"}
Oct 31 04:16:06.485895 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.485626    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"
Oct 31 04:16:06.494528 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.494512    4900 scope.go:122] "RemoveContainer" containerID="8dc4727f150855ac72e551b486df4d8f6af8e0d3965eccbdf34c2bca42ed3275"
Oct 31 04:16:06.494844 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:16:06.494823    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc4727f150855ac72e551b486df4d8f6af8e0d3965eccbdf34c2bca42ed3275\": container with ID starting with 8dc4727f150855ac72e551b486df4d8f6af8e0d3965eccbdf34c2bca42ed3275 not found: ID does not exist" containerID="8dc4727f150855ac72e551b486df4d8f6af8e0d3965eccbdf34c2bca42ed3275"
Oct 31 04:16:06.494992 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.494933    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc4727f150855ac72e551b486df4d8f6af8e0d3965eccbdf34c2bca42ed3275"} err="failed to get container status \"8dc4727f150855ac72e551b486df4d8f6af8e0d3965eccbdf34c2bca42ed3275\": rpc error: code = NotFound desc = could not find container \"8dc4727f150855ac72e551b486df4d8f6af8e0d3965eccbdf34c2bca42ed3275\": container with ID starting with 8dc4727f150855ac72e551b486df4d8f6af8e0d3965eccbdf34c2bca42ed3275 not found: ID does not exist"
Oct 31 04:16:06.495176 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.495079    4900 scope.go:122] "RemoveContainer" containerID="e80111bd79a1f16a5709f13fa5fbfb8c7df78e325cffe59edc5682cc526e81fc"
Oct 31 04:16:06.504816 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.504765    4900 scope.go:122] "RemoveContainer" containerID="e80111bd79a1f16a5709f13fa5fbfb8c7df78e325cffe59edc5682cc526e81fc"
Oct 31 04:16:06.505043 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:16:06.505027    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80111bd79a1f16a5709f13fa5fbfb8c7df78e325cffe59edc5682cc526e81fc\": container with ID starting with e80111bd79a1f16a5709f13fa5fbfb8c7df78e325cffe59edc5682cc526e81fc not found: ID does not exist" containerID="e80111bd79a1f16a5709f13fa5fbfb8c7df78e325cffe59edc5682cc526e81fc"
Oct 31 04:16:06.505099 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.505049    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80111bd79a1f16a5709f13fa5fbfb8c7df78e325cffe59edc5682cc526e81fc"} err="failed to get container status \"e80111bd79a1f16a5709f13fa5fbfb8c7df78e325cffe59edc5682cc526e81fc\": rpc error: code = NotFound desc = could not find container \"e80111bd79a1f16a5709f13fa5fbfb8c7df78e325cffe59edc5682cc526e81fc\": container with ID starting with e80111bd79a1f16a5709f13fa5fbfb8c7df78e325cffe59edc5682cc526e81fc not found: ID does not exist"
Oct 31 04:16:06.573591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573518    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573544    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573555    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573565    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjj7v\" (UniqueName: \"kubernetes.io/projected/191f5b84-f796-4f2b-8cba-a8948506b26b-kube-api-access-mjj7v\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573575    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573584    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghrvx\" (UniqueName: \"kubernetes.io/projected/86e324aa-c2e3-4524-a3f0-2d2af88b6063-kube-api-access-ghrvx\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573592    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573601    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573611    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573619    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573627    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e324aa-c2e3-4524-a3f0-2d2af88b6063-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573635    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/191f5b84-f796-4f2b-8cba-a8948506b26b-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573643    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/191f5b84-f796-4f2b-8cba-a8948506b26b-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573651    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573663    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e324aa-c2e3-4524-a3f0-2d2af88b6063-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.573953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.573671    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/191f5b84-f796-4f2b-8cba-a8948506b26b-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:06.731928 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.731893    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"]
Oct 31 04:16:06.758839 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.758000    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gh4g9"]
Oct 31 04:16:06.860858 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.859840    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"]
Oct 31 04:16:06.862374 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:06.862353    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-jg5sl"]
Oct 31 04:16:06.985916 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:16:06.985882    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:16:07.159871 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:07.159068    4900 patch_prober.go:29] interesting pod/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 04:16:07.159871 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:07.159113    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq" podUID="c0e37b79-c673-4bb9-a740-eda25789449a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 04:16:07.170023 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:07.169991    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="191f5b84-f796-4f2b-8cba-a8948506b26b" path="/var/lib/kubelet/pods/191f5b84-f796-4f2b-8cba-a8948506b26b/volumes"
Oct 31 04:16:07.170733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:07.170704    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e324aa-c2e3-4524-a3f0-2d2af88b6063" path="/var/lib/kubelet/pods/86e324aa-c2e3-4524-a3f0-2d2af88b6063/volumes"
Oct 31 04:16:07.226810 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:07.225756    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:16:08.103186 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:08.103122    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:08.159502 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:08.159472    4900 patch_prober.go:29] interesting pod/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Oct 31 04:16:08.159668 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:08.159529    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq" podUID="c0e37b79-c673-4bb9-a740-eda25789449a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Oct 31 04:16:09.164268 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:09.164236    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:16:09.165422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:09.165398    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:16:12.903560 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:12.903530    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:17.710873 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:17.710803    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:22.568424 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:22.568379    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:22.781924 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:22.781883    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:27.374091 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:27.374055    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:28.549945 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:28.549909    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:29.356682 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:29.356650    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"]
Oct 31 04:16:29.467971 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:29.467940    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"]
Oct 31 04:16:29.804953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:29.804920    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:30.009779 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:30.009747    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:32.548987 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:32.548952    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:34.335634 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.335602    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:34.356935 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.356896    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5" podUID="5bab061e-96a7-4844-a278-ce81f21d9ebf" containerName="payload-processing" containerID="cri-o://71375f038b0275fc4660cfaea50772d8bf850851dc73bcf19734189322889e2a" gracePeriod=25
Oct 31 04:16:34.468505 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.468460    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf" podUID="43d4c814-2191-4ff8-9897-d5d9fd69511a" containerName="payload-pre-processing" containerID="cri-o://38867b471fdf55f6111d896f288f53155901f61692a538516ce3592b73da5cf3" gracePeriod=25
Oct 31 04:16:34.549187 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:16:34.549146    4900 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d4c814_2191_4ff8_9897_d5d9fd69511a.slice/crio-conmon-38867b471fdf55f6111d896f288f53155901f61692a538516ce3592b73da5cf3.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 04:16:34.637407 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.637383    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"
Oct 31 04:16:34.663202 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.663170    4900 generic.go:356] "Generic (PLEG): container finished" podID="5bab061e-96a7-4844-a278-ce81f21d9ebf" containerID="71375f038b0275fc4660cfaea50772d8bf850851dc73bcf19734189322889e2a" exitCode=0
Oct 31 04:16:34.663380 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.663225    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"
Oct 31 04:16:34.663380 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.663252    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5" event={"ID":"5bab061e-96a7-4844-a278-ce81f21d9ebf","Type":"ContainerDied","Data":"71375f038b0275fc4660cfaea50772d8bf850851dc73bcf19734189322889e2a"}
Oct 31 04:16:34.663380 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.663307    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5" event={"ID":"5bab061e-96a7-4844-a278-ce81f21d9ebf","Type":"ContainerDied","Data":"14129c7ce7a0421ec5553207d6c83d1c92228c52ae6d6a36b1594347b49f4a48"}
Oct 31 04:16:34.663380 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.663333    4900 scope.go:122] "RemoveContainer" containerID="71375f038b0275fc4660cfaea50772d8bf850851dc73bcf19734189322889e2a"
Oct 31 04:16:34.665024 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.664999    4900 generic.go:356] "Generic (PLEG): container finished" podID="43d4c814-2191-4ff8-9897-d5d9fd69511a" containerID="38867b471fdf55f6111d896f288f53155901f61692a538516ce3592b73da5cf3" exitCode=0
Oct 31 04:16:34.665128 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.665061    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf" event={"ID":"43d4c814-2191-4ff8-9897-d5d9fd69511a","Type":"ContainerDied","Data":"38867b471fdf55f6111d896f288f53155901f61692a538516ce3592b73da5cf3"}
Oct 31 04:16:34.675833 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.675771    4900 scope.go:122] "RemoveContainer" containerID="71375f038b0275fc4660cfaea50772d8bf850851dc73bcf19734189322889e2a"
Oct 31 04:16:34.676245 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:16:34.676192    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71375f038b0275fc4660cfaea50772d8bf850851dc73bcf19734189322889e2a\": container with ID starting with 71375f038b0275fc4660cfaea50772d8bf850851dc73bcf19734189322889e2a not found: ID does not exist" containerID="71375f038b0275fc4660cfaea50772d8bf850851dc73bcf19734189322889e2a"
Oct 31 04:16:34.676245 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.676224    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71375f038b0275fc4660cfaea50772d8bf850851dc73bcf19734189322889e2a"} err="failed to get container status \"71375f038b0275fc4660cfaea50772d8bf850851dc73bcf19734189322889e2a\": rpc error: code = NotFound desc = could not find container \"71375f038b0275fc4660cfaea50772d8bf850851dc73bcf19734189322889e2a\": container with ID starting with 71375f038b0275fc4660cfaea50772d8bf850851dc73bcf19734189322889e2a not found: ID does not exist"
Oct 31 04:16:34.737401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.737379    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"
Oct 31 04:16:34.756574 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.756547    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bab061e-96a7-4844-a278-ce81f21d9ebf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bab061e-96a7-4844-a278-ce81f21d9ebf-plugins-config-volume\") pod \"5bab061e-96a7-4844-a278-ce81f21d9ebf\" (UID: \"5bab061e-96a7-4844-a278-ce81f21d9ebf\") "
Oct 31 04:16:34.756719 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.756674    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bab061e-96a7-4844-a278-ce81f21d9ebf-kube-api-access-skmtn\" (UniqueName: \"kubernetes.io/projected/5bab061e-96a7-4844-a278-ce81f21d9ebf-kube-api-access-skmtn\") pod \"5bab061e-96a7-4844-a278-ce81f21d9ebf\" (UID: \"5bab061e-96a7-4844-a278-ce81f21d9ebf\") "
Oct 31 04:16:34.756967 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.756943    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bab061e-96a7-4844-a278-ce81f21d9ebf-plugins-config-volume" pod "5bab061e-96a7-4844-a278-ce81f21d9ebf" (UID: "5bab061e-96a7-4844-a278-ce81f21d9ebf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:16:34.757308 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.757280    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bab061e-96a7-4844-a278-ce81f21d9ebf-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:34.758766 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.758739    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bab061e-96a7-4844-a278-ce81f21d9ebf-kube-api-access-skmtn" pod "5bab061e-96a7-4844-a278-ce81f21d9ebf" (UID: "5bab061e-96a7-4844-a278-ce81f21d9ebf"). InnerVolumeSpecName "kube-api-access-skmtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:34.858482 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.858402    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d4c814-2191-4ff8-9897-d5d9fd69511a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d4c814-2191-4ff8-9897-d5d9fd69511a-plugins-config-volume\") pod \"43d4c814-2191-4ff8-9897-d5d9fd69511a\" (UID: \"43d4c814-2191-4ff8-9897-d5d9fd69511a\") "
Oct 31 04:16:34.858640 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.858508    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d4c814-2191-4ff8-9897-d5d9fd69511a-kube-api-access-h79kv\" (UniqueName: \"kubernetes.io/projected/43d4c814-2191-4ff8-9897-d5d9fd69511a-kube-api-access-h79kv\") pod \"43d4c814-2191-4ff8-9897-d5d9fd69511a\" (UID: \"43d4c814-2191-4ff8-9897-d5d9fd69511a\") "
Oct 31 04:16:34.858777 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.858756    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d4c814-2191-4ff8-9897-d5d9fd69511a-plugins-config-volume" pod "43d4c814-2191-4ff8-9897-d5d9fd69511a" (UID: "43d4c814-2191-4ff8-9897-d5d9fd69511a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:16:34.859082 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.859058    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skmtn\" (UniqueName: \"kubernetes.io/projected/5bab061e-96a7-4844-a278-ce81f21d9ebf-kube-api-access-skmtn\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:34.859082 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.859075    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43d4c814-2191-4ff8-9897-d5d9fd69511a-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:34.860585 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.860565    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d4c814-2191-4ff8-9897-d5d9fd69511a-kube-api-access-h79kv" pod "43d4c814-2191-4ff8-9897-d5d9fd69511a" (UID: "43d4c814-2191-4ff8-9897-d5d9fd69511a"). InnerVolumeSpecName "kube-api-access-h79kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:34.960202 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:34.960170    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h79kv\" (UniqueName: \"kubernetes.io/projected/43d4c814-2191-4ff8-9897-d5d9fd69511a-kube-api-access-h79kv\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:35.082731 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:35.082698    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"]
Oct 31 04:16:35.088135 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:35.088111    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-83f052-7dcf7f9c85-7gwt5"]
Oct 31 04:16:35.154167 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:35.154088    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bab061e-96a7-4844-a278-ce81f21d9ebf" path="/var/lib/kubelet/pods/5bab061e-96a7-4844-a278-ce81f21d9ebf/volumes"
Oct 31 04:16:35.673877 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:35.673842    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf" event={"ID":"43d4c814-2191-4ff8-9897-d5d9fd69511a","Type":"ContainerDied","Data":"c6a255adfdde12e78fd014fd681cc12130e3db28fb23287b231cc891c05a03b9"}
Oct 31 04:16:35.673877 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:35.673868    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"
Oct 31 04:16:35.674456 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:35.673914    4900 scope.go:122] "RemoveContainer" containerID="38867b471fdf55f6111d896f288f53155901f61692a538516ce3592b73da5cf3"
Oct 31 04:16:35.838377 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:35.838349    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"]
Oct 31 04:16:35.888202 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:35.888174    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-83f052-fd69f94c9-q9qrf"]
Oct 31 04:16:36.984834 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:16:36.984770    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:16:37.154316 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:37.154282    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d4c814-2191-4ff8-9897-d5d9fd69511a" path="/var/lib/kubelet/pods/43d4c814-2191-4ff8-9897-d5d9fd69511a/volumes"
Oct 31 04:16:37.736253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:37.736225    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:38.919174 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:38.919142    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:42.498888 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:42.498849    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:44.274945 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:44.274870    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"]
Oct 31 04:16:44.613465 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:44.613372    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:44.827186 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:44.827150    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:47.773904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:47.773866    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:49.278877 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.278806    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8" podUID="b9b6731b-e6fe-4b7d-b349-18acfb435f95" containerName="payload-pre-processing" containerID="cri-o://83d091851d8fc2e5d0359330bbf82e6c95985ceeb7636ceff6811447f1d3b0db" gracePeriod=25
Oct 31 04:16:49.543137 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.543081    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"
Oct 31 04:16:49.652819 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.652768    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b6731b-e6fe-4b7d-b349-18acfb435f95-kube-api-access-75j79\" (UniqueName: \"kubernetes.io/projected/b9b6731b-e6fe-4b7d-b349-18acfb435f95-kube-api-access-75j79\") pod \"b9b6731b-e6fe-4b7d-b349-18acfb435f95\" (UID: \"b9b6731b-e6fe-4b7d-b349-18acfb435f95\") "
Oct 31 04:16:49.652975 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.652950    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9b6731b-e6fe-4b7d-b349-18acfb435f95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b6731b-e6fe-4b7d-b349-18acfb435f95-plugins-config-volume\") pod \"b9b6731b-e6fe-4b7d-b349-18acfb435f95\" (UID: \"b9b6731b-e6fe-4b7d-b349-18acfb435f95\") "
Oct 31 04:16:49.653304 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.653278    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b6731b-e6fe-4b7d-b349-18acfb435f95-plugins-config-volume" pod "b9b6731b-e6fe-4b7d-b349-18acfb435f95" (UID: "b9b6731b-e6fe-4b7d-b349-18acfb435f95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:16:49.655223 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.655199    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b6731b-e6fe-4b7d-b349-18acfb435f95-kube-api-access-75j79" pod "b9b6731b-e6fe-4b7d-b349-18acfb435f95" (UID: "b9b6731b-e6fe-4b7d-b349-18acfb435f95"). InnerVolumeSpecName "kube-api-access-75j79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:16:49.754874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.754839    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75j79\" (UniqueName: \"kubernetes.io/projected/b9b6731b-e6fe-4b7d-b349-18acfb435f95-kube-api-access-75j79\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:49.754874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.754875    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9b6731b-e6fe-4b7d-b349-18acfb435f95-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:16:49.758453 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.758424    4900 generic.go:356] "Generic (PLEG): container finished" podID="b9b6731b-e6fe-4b7d-b349-18acfb435f95" containerID="83d091851d8fc2e5d0359330bbf82e6c95985ceeb7636ceff6811447f1d3b0db" exitCode=0
Oct 31 04:16:49.758607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.758475    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"
Oct 31 04:16:49.758607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.758511    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8" event={"ID":"b9b6731b-e6fe-4b7d-b349-18acfb435f95","Type":"ContainerDied","Data":"83d091851d8fc2e5d0359330bbf82e6c95985ceeb7636ceff6811447f1d3b0db"}
Oct 31 04:16:49.758607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.758566    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8" event={"ID":"b9b6731b-e6fe-4b7d-b349-18acfb435f95","Type":"ContainerDied","Data":"032441eb5dbf88b6881f5b23275cfee1f1f81123385af1ff28160b463261b787"}
Oct 31 04:16:49.758607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.758589    4900 scope.go:122] "RemoveContainer" containerID="83d091851d8fc2e5d0359330bbf82e6c95985ceeb7636ceff6811447f1d3b0db"
Oct 31 04:16:49.772430 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.772396    4900 scope.go:122] "RemoveContainer" containerID="83d091851d8fc2e5d0359330bbf82e6c95985ceeb7636ceff6811447f1d3b0db"
Oct 31 04:16:49.772947 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:16:49.772917    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d091851d8fc2e5d0359330bbf82e6c95985ceeb7636ceff6811447f1d3b0db\": container with ID starting with 83d091851d8fc2e5d0359330bbf82e6c95985ceeb7636ceff6811447f1d3b0db not found: ID does not exist" containerID="83d091851d8fc2e5d0359330bbf82e6c95985ceeb7636ceff6811447f1d3b0db"
Oct 31 04:16:49.773082 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.772958    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d091851d8fc2e5d0359330bbf82e6c95985ceeb7636ceff6811447f1d3b0db"} err="failed to get container status \"83d091851d8fc2e5d0359330bbf82e6c95985ceeb7636ceff6811447f1d3b0db\": rpc error: code = NotFound desc = could not find container \"83d091851d8fc2e5d0359330bbf82e6c95985ceeb7636ceff6811447f1d3b0db\": container with ID starting with 83d091851d8fc2e5d0359330bbf82e6c95985ceeb7636ceff6811447f1d3b0db not found: ID does not exist"
Oct 31 04:16:49.924352 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.924268    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"]
Oct 31 04:16:49.929187 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:49.929160    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e53098-6d5f8d7879-68rm8"]
Oct 31 04:16:50.391015 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:50.390975    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:51.152733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:51.152705    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b6731b-e6fe-4b7d-b349-18acfb435f95" path="/var/lib/kubelet/pods/b9b6731b-e6fe-4b7d-b349-18acfb435f95/volumes"
Oct 31 04:16:52.827983 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:52.827951    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:16:57.878248 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:16:57.878202    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:02.684030 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:02.683992    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:05.990136 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:05.990100    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:06.717668 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:06.717635    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"]
Oct 31 04:17:06.717877 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:06.717852    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq" podUID="0cbb8e5d-5188-4305-85dc-a7ee1b3b08df" containerName="maas-api" containerID="cri-o://45d2d4051f6ed02a15cbf856afaa6d6c5f8e87ecb08aaea16911f8eddcbe196f" gracePeriod=30
Oct 31 04:17:06.865714 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:06.865681    4900 generic.go:356] "Generic (PLEG): container finished" podID="0cbb8e5d-5188-4305-85dc-a7ee1b3b08df" containerID="45d2d4051f6ed02a15cbf856afaa6d6c5f8e87ecb08aaea16911f8eddcbe196f" exitCode=0
Oct 31 04:17:06.865914 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:06.865749    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq" event={"ID":"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df","Type":"ContainerDied","Data":"45d2d4051f6ed02a15cbf856afaa6d6c5f8e87ecb08aaea16911f8eddcbe196f"}
Oct 31 04:17:06.989225 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:17:06.989190    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:17:06.992533 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:06.992507    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:17:07.171244 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.171215    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-openshift-service-ca\") pod \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") "
Oct 31 04:17:07.171446 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.171376    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-maas-api-tls\") pod \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") "
Oct 31 04:17:07.171446 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.171443    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-metrics-certs\") pod \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") "
Oct 31 04:17:07.171570 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.171543    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-kube-api-access-5n7s5\" (UniqueName: \"kubernetes.io/projected/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-kube-api-access-5n7s5\") pod \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\" (UID: \"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df\") "
Oct 31 04:17:07.171654 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.171621    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-openshift-service-ca" pod "0cbb8e5d-5188-4305-85dc-a7ee1b3b08df" (UID: "0cbb8e5d-5188-4305-85dc-a7ee1b3b08df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:17:07.172160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.172139    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:07.174002 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.173978    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-metrics-certs" pod "0cbb8e5d-5188-4305-85dc-a7ee1b3b08df" (UID: "0cbb8e5d-5188-4305-85dc-a7ee1b3b08df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:17:07.174100 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.174058    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-maas-api-tls" pod "0cbb8e5d-5188-4305-85dc-a7ee1b3b08df" (UID: "0cbb8e5d-5188-4305-85dc-a7ee1b3b08df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:17:07.174362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.174344    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-kube-api-access-5n7s5" pod "0cbb8e5d-5188-4305-85dc-a7ee1b3b08df" (UID: "0cbb8e5d-5188-4305-85dc-a7ee1b3b08df"). InnerVolumeSpecName "kube-api-access-5n7s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:17:07.273766 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.273729    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:07.273766 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.273757    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:07.273766 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.273769    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n7s5\" (UniqueName: \"kubernetes.io/projected/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df-kube-api-access-5n7s5\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:07.479754 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.479704    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:07.872404 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.872378    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"
Oct 31 04:17:07.872581 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.872418    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq" event={"ID":"0cbb8e5d-5188-4305-85dc-a7ee1b3b08df","Type":"ContainerDied","Data":"80249ce04c1dc9c11b92be15266d8a8ab96e35ee67a03d446c31eac160d60f80"}
Oct 31 04:17:07.872581 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:07.872483    4900 scope.go:122] "RemoveContainer" containerID="45d2d4051f6ed02a15cbf856afaa6d6c5f8e87ecb08aaea16911f8eddcbe196f"
Oct 31 04:17:08.086311 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:08.086271    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"]
Oct 31 04:17:08.136103 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:08.136027    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f0e007-6d7869cbb4-jr8lq"]
Oct 31 04:17:08.572187 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:08.572155    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:08.796280 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:08.796250    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:09.154909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:09.154875    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cbb8e5d-5188-4305-85dc-a7ee1b3b08df" path="/var/lib/kubelet/pods/0cbb8e5d-5188-4305-85dc-a7ee1b3b08df/volumes"
Oct 31 04:17:09.818223 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:09.818188    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:10.032450 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:10.032408    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:11.454129 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:11.454098    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:11.989174 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:11.989140    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:12.812444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:12.812410    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:13.457879 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.457849    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"]
Oct 31 04:17:13.458413 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458399    4900 cpu_manager.go:414] "containerMap: removing container" podUID="a2146db6-ed0b-4776-930f-e1f915526586" containerName="cleanup"
Oct 31 04:17:13.458458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458414    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2146db6-ed0b-4776-930f-e1f915526586" containerName="cleanup"
Oct 31 04:17:13.458458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458429    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0cbb8e5d-5188-4305-85dc-a7ee1b3b08df" containerName="maas-api"
Oct 31 04:17:13.458458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458434    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cbb8e5d-5188-4305-85dc-a7ee1b3b08df" containerName="maas-api"
Oct 31 04:17:13.458458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458444    4900 cpu_manager.go:414] "containerMap: removing container" podUID="5bab061e-96a7-4844-a278-ce81f21d9ebf" containerName="payload-processing"
Oct 31 04:17:13.458458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458449    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bab061e-96a7-4844-a278-ce81f21d9ebf" containerName="payload-processing"
Oct 31 04:17:13.458458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458455    4900 cpu_manager.go:414] "containerMap: removing container" podUID="86e324aa-c2e3-4524-a3f0-2d2af88b6063" containerName="istio-proxy"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458462    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e324aa-c2e3-4524-a3f0-2d2af88b6063" containerName="istio-proxy"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458472    4900 cpu_manager.go:414] "containerMap: removing container" podUID="43d4c814-2191-4ff8-9897-d5d9fd69511a" containerName="payload-pre-processing"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458478    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d4c814-2191-4ff8-9897-d5d9fd69511a" containerName="payload-pre-processing"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458488    4900 cpu_manager.go:414] "containerMap: removing container" podUID="191f5b84-f796-4f2b-8cba-a8948506b26b" containerName="istio-proxy"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458492    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191f5b84-f796-4f2b-8cba-a8948506b26b" containerName="istio-proxy"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458500    4900 cpu_manager.go:414] "containerMap: removing container" podUID="f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473" containerName="payload-processing"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458504    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473" containerName="payload-processing"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458516    4900 cpu_manager.go:414] "containerMap: removing container" podUID="713a4870-e144-4960-bbff-0c4e623ba57c" containerName="cleanup"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458520    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713a4870-e144-4960-bbff-0c4e623ba57c" containerName="cleanup"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458529    4900 cpu_manager.go:414] "containerMap: removing container" podUID="b9b6731b-e6fe-4b7d-b349-18acfb435f95" containerName="payload-pre-processing"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458534    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b6731b-e6fe-4b7d-b349-18acfb435f95" containerName="payload-pre-processing"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458546    4900 cpu_manager.go:414] "containerMap: removing container" podUID="695cf523-b73f-4e3d-999e-eb0b36ad3ed5" containerName="payload-pre-processing"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458552    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695cf523-b73f-4e3d-999e-eb0b36ad3ed5" containerName="payload-pre-processing"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458559    4900 cpu_manager.go:414] "containerMap: removing container" podUID="e4447ca2-888d-4950-b75d-b6149d71791a" containerName="cleanup"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458564    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4447ca2-888d-4950-b75d-b6149d71791a" containerName="cleanup"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458587    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8797d751-7c89-4eb8-aaf9-abf0c81efcca" containerName="cleanup"
Oct 31 04:17:13.458653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458592    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8797d751-7c89-4eb8-aaf9-abf0c81efcca" containerName="cleanup"
Oct 31 04:17:13.459207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458665    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a9e5b0-9d34-468d-a70a-304974436a9b" containerName="cleanup"
Oct 31 04:17:13.459207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458677    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8797d751-7c89-4eb8-aaf9-abf0c81efcca" containerName="cleanup"
Oct 31 04:17:13.459207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458684    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecfcb4c4-007c-4acc-a20c-092b12cfc44b" containerName="cleanup"
Oct 31 04:17:13.459207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458690    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cbb8e5d-5188-4305-85dc-a7ee1b3b08df" containerName="maas-api"
Oct 31 04:17:13.459207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458696    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bab061e-96a7-4844-a278-ce81f21d9ebf" containerName="payload-processing"
Oct 31 04:17:13.459207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458702    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e324aa-c2e3-4524-a3f0-2d2af88b6063" containerName="istio-proxy"
Oct 31 04:17:13.459207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458707    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d4c814-2191-4ff8-9897-d5d9fd69511a" containerName="payload-pre-processing"
Oct 31 04:17:13.459207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458712    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="191f5b84-f796-4f2b-8cba-a8948506b26b" containerName="istio-proxy"
Oct 31 04:17:13.459207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458718    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6bbb717-ecfb-4dd5-ae7a-82f80d2d6473" containerName="payload-processing"
Oct 31 04:17:13.459207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458723    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="713a4870-e144-4960-bbff-0c4e623ba57c" containerName="cleanup"
Oct 31 04:17:13.459207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458731    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b6731b-e6fe-4b7d-b349-18acfb435f95" containerName="payload-pre-processing"
Oct 31 04:17:13.459207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458737    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b45f245-d9e1-4476-bdb9-4a840c078444" containerName="cleanup"
Oct 31 04:17:13.459207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458742    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="695cf523-b73f-4e3d-999e-eb0b36ad3ed5" containerName="payload-pre-processing"
Oct 31 04:17:13.459207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.458747    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4447ca2-888d-4950-b75d-b6149d71791a" containerName="cleanup"
Oct 31 04:17:13.462118 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.462099    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:13.514941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.514912    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"]
Oct 31 04:17:13.548277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.548251    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:13.548420 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.548302    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-metrics-certs\") pod \"maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:13.548420 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.548321    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb9xs\" (UniqueName: \"kubernetes.io/projected/3e59d7a0-72ed-4065-9538-cc15b531b5fe-kube-api-access-bb9xs\") pod \"maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:13.548504 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.548420    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e59d7a0-72ed-4065-9538-cc15b531b5fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:13.561603 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.561583    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c42c43-serving-cert\""
Oct 31 04:17:13.649261 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.649221    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-metrics-certs\") pod \"maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:13.649261 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.649260    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb9xs\" (UniqueName: \"kubernetes.io/projected/3e59d7a0-72ed-4065-9538-cc15b531b5fe-kube-api-access-bb9xs\") pod \"maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:13.649502 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.649292    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e59d7a0-72ed-4065-9538-cc15b531b5fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:13.649567 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.649554    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:13.649743 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:17:13.649722    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-serving-cert: secret "maas-api-e2e-inf-a-c42c43-serving-cert" not found
Oct 31 04:17:13.649977 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:17:13.649962    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-maas-api-tls podName:3e59d7a0-72ed-4065-9538-cc15b531b5fe nodeName:}" failed. No retries permitted until 2026-10-31 04:17:14.149940204 +0000 UTC m=+4187.116273549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-maas-api-tls") pod "maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c" (UID: "3e59d7a0-72ed-4065-9538-cc15b531b5fe") : secret "maas-api-e2e-inf-a-c42c43-serving-cert" not found
Oct 31 04:17:13.650639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.650616    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e59d7a0-72ed-4065-9538-cc15b531b5fe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:13.652210 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.652189    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-metrics-certs\") pod \"maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:13.706642 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.706614    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb9xs\" (UniqueName: \"kubernetes.io/projected/3e59d7a0-72ed-4065-9538-cc15b531b5fe-kube-api-access-bb9xs\") pod \"maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:13.775950 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.775913    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:13.980733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:13.980679    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:14.157257 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:14.157174    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:14.159617 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:14.159595    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-maas-api-tls\") pod \"maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:14.375459 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:14.375424    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:14.747523 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:17:14.747483    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e59d7a0_72ed_4065_9538_cc15b531b5fe.slice/crio-ff1aff81cda83cc412dadefb4a3bbad1f7c0f315793cb6fc3a6dab8f08f2e8f3 WatchSource:0}: Error finding container ff1aff81cda83cc412dadefb4a3bbad1f7c0f315793cb6fc3a6dab8f08f2e8f3: Status 404 returned error can't find the container with id ff1aff81cda83cc412dadefb4a3bbad1f7c0f315793cb6fc3a6dab8f08f2e8f3
Oct 31 04:17:14.749035 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:14.749015    4900 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:17:14.750022 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:14.750001    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"]
Oct 31 04:17:14.924478 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:14.924446    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c" event={"ID":"3e59d7a0-72ed-4065-9538-cc15b531b5fe","Type":"ContainerStarted","Data":"ff1aff81cda83cc412dadefb4a3bbad1f7c0f315793cb6fc3a6dab8f08f2e8f3"}
Oct 31 04:17:15.944437 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:15.944398    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c" event={"ID":"3e59d7a0-72ed-4065-9538-cc15b531b5fe","Type":"ContainerStarted","Data":"6855f932a2ceedcdcaa9bda287c74bfad763b5df2e07a22115aeb99def30d8dd"}
Oct 31 04:17:15.944879 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:15.944583    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:16.067009 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:16.066917    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c" podStartSLOduration=2.062500564 podStartE2EDuration="3.066903759s" totalImagesPullingTime="1.004403195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:17:16.066749871 +0000 UTC m=+4189.033083226" watchObservedRunningTime="2026-10-31 04:17:16.066903759 +0000 UTC m=+4189.033237102"
Oct 31 04:17:18.667190 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:18.667154    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:19.312065 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:19.312012    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:21.955055 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:21.955024    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:17:22.577832 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:22.577768    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"]
Oct 31 04:17:22.916692 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:22.914291    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:23.117633 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:23.117590    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:23.867747 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:23.867710    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:25.931586 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:25.931551    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:27.579109 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:27.579059    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r" podUID="70848e1b-1960-4cf9-bf4e-fabd84a3e16d" containerName="payload-pre-processing" containerID="cri-o://1fff83e89c4b02bf79f16ca1491cf37d13e07c0c7e7da44c3ec828f5c7203913" gracePeriod=25
Oct 31 04:17:27.839573 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:27.839511    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"
Oct 31 04:17:27.931848 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:27.931774    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70848e1b-1960-4cf9-bf4e-fabd84a3e16d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70848e1b-1960-4cf9-bf4e-fabd84a3e16d-plugins-config-volume\") pod \"70848e1b-1960-4cf9-bf4e-fabd84a3e16d\" (UID: \"70848e1b-1960-4cf9-bf4e-fabd84a3e16d\") "
Oct 31 04:17:27.932057 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:27.932000    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70848e1b-1960-4cf9-bf4e-fabd84a3e16d-kube-api-access-bj5ch\" (UniqueName: \"kubernetes.io/projected/70848e1b-1960-4cf9-bf4e-fabd84a3e16d-kube-api-access-bj5ch\") pod \"70848e1b-1960-4cf9-bf4e-fabd84a3e16d\" (UID: \"70848e1b-1960-4cf9-bf4e-fabd84a3e16d\") "
Oct 31 04:17:27.932252 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:27.932228    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70848e1b-1960-4cf9-bf4e-fabd84a3e16d-plugins-config-volume" pod "70848e1b-1960-4cf9-bf4e-fabd84a3e16d" (UID: "70848e1b-1960-4cf9-bf4e-fabd84a3e16d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:17:27.932558 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:27.932530    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70848e1b-1960-4cf9-bf4e-fabd84a3e16d-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:27.934164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:27.934130    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70848e1b-1960-4cf9-bf4e-fabd84a3e16d-kube-api-access-bj5ch" pod "70848e1b-1960-4cf9-bf4e-fabd84a3e16d" (UID: "70848e1b-1960-4cf9-bf4e-fabd84a3e16d"). InnerVolumeSpecName "kube-api-access-bj5ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:17:28.022382 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:28.022349    4900 generic.go:356] "Generic (PLEG): container finished" podID="70848e1b-1960-4cf9-bf4e-fabd84a3e16d" containerID="1fff83e89c4b02bf79f16ca1491cf37d13e07c0c7e7da44c3ec828f5c7203913" exitCode=0
Oct 31 04:17:28.022560 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:28.022392    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"
Oct 31 04:17:28.022560 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:28.022426    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r" event={"ID":"70848e1b-1960-4cf9-bf4e-fabd84a3e16d","Type":"ContainerDied","Data":"1fff83e89c4b02bf79f16ca1491cf37d13e07c0c7e7da44c3ec828f5c7203913"}
Oct 31 04:17:28.022560 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:28.022479    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r" event={"ID":"70848e1b-1960-4cf9-bf4e-fabd84a3e16d","Type":"ContainerDied","Data":"c680826195cd1d289de63527b91a7c39868f503b64f3858c36dcc67ba32daaab"}
Oct 31 04:17:28.022560 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:28.022500    4900 scope.go:122] "RemoveContainer" containerID="1fff83e89c4b02bf79f16ca1491cf37d13e07c0c7e7da44c3ec828f5c7203913"
Oct 31 04:17:28.032997 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:28.032953    4900 scope.go:122] "RemoveContainer" containerID="1fff83e89c4b02bf79f16ca1491cf37d13e07c0c7e7da44c3ec828f5c7203913"
Oct 31 04:17:28.033288 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:17:28.033260    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fff83e89c4b02bf79f16ca1491cf37d13e07c0c7e7da44c3ec828f5c7203913\": container with ID starting with 1fff83e89c4b02bf79f16ca1491cf37d13e07c0c7e7da44c3ec828f5c7203913 not found: ID does not exist" containerID="1fff83e89c4b02bf79f16ca1491cf37d13e07c0c7e7da44c3ec828f5c7203913"
Oct 31 04:17:28.033376 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:28.033301    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fff83e89c4b02bf79f16ca1491cf37d13e07c0c7e7da44c3ec828f5c7203913"} err="failed to get container status \"1fff83e89c4b02bf79f16ca1491cf37d13e07c0c7e7da44c3ec828f5c7203913\": rpc error: code = NotFound desc = could not find container \"1fff83e89c4b02bf79f16ca1491cf37d13e07c0c7e7da44c3ec828f5c7203913\": container with ID starting with 1fff83e89c4b02bf79f16ca1491cf37d13e07c0c7e7da44c3ec828f5c7203913 not found: ID does not exist"
Oct 31 04:17:28.033862 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:28.033839    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj5ch\" (UniqueName: \"kubernetes.io/projected/70848e1b-1960-4cf9-bf4e-fabd84a3e16d-kube-api-access-bj5ch\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:28.186301 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:28.186198    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"]
Oct 31 04:17:28.188730 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:28.188684    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-243a29-595f587ccf-dp87r"]
Oct 31 04:17:28.907363 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:28.907339    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:17:28.918166 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:28.918143    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:17:29.153229 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:29.153187    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70848e1b-1960-4cf9-bf4e-fabd84a3e16d" path="/var/lib/kubelet/pods/70848e1b-1960-4cf9-bf4e-fabd84a3e16d/volumes"
Oct 31 04:17:30.706279 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:30.706240    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:32.564324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:32.564289    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:32.775551 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:32.775517    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:33.125163 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:33.125103    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:33.327106 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:33.327074    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:35.724530 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:35.724498    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:35.928602 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:35.928571    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:36.985875 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:17:36.985834    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:17:38.112071 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:38.112033    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:38.960852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:38.960776    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:39.175736 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:39.175701    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:42.728567 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:42.728529    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:44.669152 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:44.669114    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:47.687363 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:47.687333    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:47.909846 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:47.909757    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:49.241950 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:49.241917    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:49.699805 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:49.699696    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"]
Oct 31 04:17:49.700034 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:49.699969    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm" podUID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerName="istio-proxy" containerID="cri-o://989718ee4bda586f10574b4fa4cf340d5c274f5f355fc9d19bfef17b674aa175" gracePeriod=30
Oct 31 04:17:49.761842 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:49.761811    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:49.975553 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:49.975454    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:50.565203 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:50.565172    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:54.969349 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:54.969287    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:17:55.088050 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.088015    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-workload-certs\") pod \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") "
Oct 31 04:17:55.088050 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.088058    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istiod-ca-cert\") pod \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") "
Oct 31 04:17:55.088324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.088094    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-podinfo\") pod \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") "
Oct 31 04:17:55.088324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.088176    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-data\") pod \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") "
Oct 31 04:17:55.088324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.088250    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-envoy\") pod \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") "
Oct 31 04:17:55.088324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.088284    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-workload-socket\") pod \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") "
Oct 31 04:17:55.088558 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.088339    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-workload-certs" pod "7af499b8-dbd8-4a13-b009-af4aff66fc7b" (UID: "7af499b8-dbd8-4a13-b009-af4aff66fc7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:55.088558 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.088358    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-token\" (UniqueName: \"kubernetes.io/projected/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-token\") pod \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") "
Oct 31 04:17:55.088558 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.088461    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7af499b8-dbd8-4a13-b009-af4aff66fc7b-kube-api-access-b6xjq\" (UniqueName: \"kubernetes.io/projected/7af499b8-dbd8-4a13-b009-af4aff66fc7b-kube-api-access-b6xjq\") pod \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") "
Oct 31 04:17:55.088558 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.088512    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-data" pod "7af499b8-dbd8-4a13-b009-af4aff66fc7b" (UID: "7af499b8-dbd8-4a13-b009-af4aff66fc7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:55.088558 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.088549    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-credential-socket\") pod \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\" (UID: \"7af499b8-dbd8-4a13-b009-af4aff66fc7b\") "
Oct 31 04:17:55.088857 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.088546    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istiod-ca-cert" pod "7af499b8-dbd8-4a13-b009-af4aff66fc7b" (UID: "7af499b8-dbd8-4a13-b009-af4aff66fc7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:17:55.088857 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.088632    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-workload-socket" pod "7af499b8-dbd8-4a13-b009-af4aff66fc7b" (UID: "7af499b8-dbd8-4a13-b009-af4aff66fc7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:55.088969 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.088878    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-credential-socket" pod "7af499b8-dbd8-4a13-b009-af4aff66fc7b" (UID: "7af499b8-dbd8-4a13-b009-af4aff66fc7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:55.089236 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.089215    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:55.089287 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.089242    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:55.089287 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.089259    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:55.089287 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.089274    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:55.089401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.089289    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:55.090604 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.090580    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-token" pod "7af499b8-dbd8-4a13-b009-af4aff66fc7b" (UID: "7af499b8-dbd8-4a13-b009-af4aff66fc7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:17:55.090738 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.090614    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-envoy" pod "7af499b8-dbd8-4a13-b009-af4aff66fc7b" (UID: "7af499b8-dbd8-4a13-b009-af4aff66fc7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:17:55.090956 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.090929    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-podinfo" pod "7af499b8-dbd8-4a13-b009-af4aff66fc7b" (UID: "7af499b8-dbd8-4a13-b009-af4aff66fc7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:17:55.090956 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.090946    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af499b8-dbd8-4a13-b009-af4aff66fc7b-kube-api-access-b6xjq" pod "7af499b8-dbd8-4a13-b009-af4aff66fc7b" (UID: "7af499b8-dbd8-4a13-b009-af4aff66fc7b"). InnerVolumeSpecName "kube-api-access-b6xjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:17:55.190257 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.190220    4900 generic.go:356] "Generic (PLEG): container finished" podID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerID="989718ee4bda586f10574b4fa4cf340d5c274f5f355fc9d19bfef17b674aa175" exitCode=0
Oct 31 04:17:55.190422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.190274    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"
Oct 31 04:17:55.190422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.190285    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm" event={"ID":"7af499b8-dbd8-4a13-b009-af4aff66fc7b","Type":"ContainerDied","Data":"989718ee4bda586f10574b4fa4cf340d5c274f5f355fc9d19bfef17b674aa175"}
Oct 31 04:17:55.190422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.190339    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm" event={"ID":"7af499b8-dbd8-4a13-b009-af4aff66fc7b","Type":"ContainerDied","Data":"23076b2b18fba5ade6f781e41d65c8857e8892563ac6f27cde7951ad3c142f91"}
Oct 31 04:17:55.190422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.190371    4900 scope.go:122] "RemoveContainer" containerID="989718ee4bda586f10574b4fa4cf340d5c274f5f355fc9d19bfef17b674aa175"
Oct 31 04:17:55.190625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.190474    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:55.190625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.190600    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:55.190724 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.190625    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6xjq\" (UniqueName: \"kubernetes.io/projected/7af499b8-dbd8-4a13-b009-af4aff66fc7b-kube-api-access-b6xjq\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:55.190724 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.190643    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7af499b8-dbd8-4a13-b009-af4aff66fc7b-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:17:55.200646 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.200624    4900 scope.go:122] "RemoveContainer" containerID="d9966e8ef70e83ce0d49a439548259d433ffdde2d3e511c4de1cf6c65be1849f"
Oct 31 04:17:55.209654 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.209634    4900 scope.go:122] "RemoveContainer" containerID="989718ee4bda586f10574b4fa4cf340d5c274f5f355fc9d19bfef17b674aa175"
Oct 31 04:17:55.209968 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:17:55.209945    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989718ee4bda586f10574b4fa4cf340d5c274f5f355fc9d19bfef17b674aa175\": container with ID starting with 989718ee4bda586f10574b4fa4cf340d5c274f5f355fc9d19bfef17b674aa175 not found: ID does not exist" containerID="989718ee4bda586f10574b4fa4cf340d5c274f5f355fc9d19bfef17b674aa175"
Oct 31 04:17:55.210026 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.209981    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989718ee4bda586f10574b4fa4cf340d5c274f5f355fc9d19bfef17b674aa175"} err="failed to get container status \"989718ee4bda586f10574b4fa4cf340d5c274f5f355fc9d19bfef17b674aa175\": rpc error: code = NotFound desc = could not find container \"989718ee4bda586f10574b4fa4cf340d5c274f5f355fc9d19bfef17b674aa175\": container with ID starting with 989718ee4bda586f10574b4fa4cf340d5c274f5f355fc9d19bfef17b674aa175 not found: ID does not exist"
Oct 31 04:17:55.210026 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.209999    4900 scope.go:122] "RemoveContainer" containerID="d9966e8ef70e83ce0d49a439548259d433ffdde2d3e511c4de1cf6c65be1849f"
Oct 31 04:17:55.210270 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:17:55.210249    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9966e8ef70e83ce0d49a439548259d433ffdde2d3e511c4de1cf6c65be1849f\": container with ID starting with d9966e8ef70e83ce0d49a439548259d433ffdde2d3e511c4de1cf6c65be1849f not found: ID does not exist" containerID="d9966e8ef70e83ce0d49a439548259d433ffdde2d3e511c4de1cf6c65be1849f"
Oct 31 04:17:55.210345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.210279    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9966e8ef70e83ce0d49a439548259d433ffdde2d3e511c4de1cf6c65be1849f"} err="failed to get container status \"d9966e8ef70e83ce0d49a439548259d433ffdde2d3e511c4de1cf6c65be1849f\": rpc error: code = NotFound desc = could not find container \"d9966e8ef70e83ce0d49a439548259d433ffdde2d3e511c4de1cf6c65be1849f\": container with ID starting with d9966e8ef70e83ce0d49a439548259d433ffdde2d3e511c4de1cf6c65be1849f not found: ID does not exist"
Oct 31 04:17:55.229294 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.229208    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:55.356277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.356239    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"]
Oct 31 04:17:55.361306 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:55.361274    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-83f052-openshift-default-586d975ff-mr2zm"]
Oct 31 04:17:56.474059 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:56.474022    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:57.153319 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:57.153288    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" path="/var/lib/kubelet/pods/7af499b8-dbd8-4a13-b009-af4aff66fc7b/volumes"
Oct 31 04:17:57.514382 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:57.514334    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:57.725417 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:57.725386    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:17:59.846896 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:17:59.846857    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:01.905091 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:01.905059    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:02.159717 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:02.159559    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:05.862764 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:05.862725    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:06.069969 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:06.069930    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:06.373078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:06.373033    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:06.941280 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:06.941241    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:06.987449 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:18:06.987408    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:18:07.505320 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:07.505283    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:08.048954 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:08.048921    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:08.304131 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:08.304096    4900 generic.go:356] "Generic (PLEG): container finished" podID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerID="a096ad3d8e428037aaf1a67313426834d9b7411c2a2119c8db836703ff6f879b" exitCode=0
Oct 31 04:18:08.304330 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:08.304151    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r" event={"ID":"6ed827b7-7ac6-412d-ade0-70ce0d8cc399","Type":"ContainerDied","Data":"a096ad3d8e428037aaf1a67313426834d9b7411c2a2119c8db836703ff6f879b"}
Oct 31 04:18:08.304624 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:08.304597    4900 scope.go:122] "RemoveContainer" containerID="a096ad3d8e428037aaf1a67313426834d9b7411c2a2119c8db836703ff6f879b"
Oct 31 04:18:08.306153 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:08.306127    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:18:08.306227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:08.306198    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:18:08.306273 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:08.306231    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:18:08.817777 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:08.817739    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:09.312349 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:09.312312    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r" event={"ID":"6ed827b7-7ac6-412d-ade0-70ce0d8cc399","Type":"ContainerStarted","Data":"ccccaebf035067f2154410fd2216959e1d515fae4c6d59a3563251ac02c0b969"}
Oct 31 04:18:09.704444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:09.704365    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:18:09.705602 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:09.705572    4900 patch_prober.go:29] interesting pod/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 04:18:09.705748 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:09.705639    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r" podUID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 04:18:10.703932 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:10.703894    4900 patch_prober.go:29] interesting pod/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 04:18:10.704456 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:10.703967    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r" podUID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 04:18:11.704334 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:11.704294    4900 patch_prober.go:29] interesting pod/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 04:18:11.704858 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:11.704353    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r" podUID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 04:18:12.419756 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:12.419717    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:12.703734 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:12.703656    4900 patch_prober.go:29] interesting pod/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 04:18:12.703734 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:12.703706    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r" podUID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 04:18:13.306717 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:13.306683    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:18:13.704432 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:13.704341    4900 patch_prober.go:29] interesting pod/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Oct 31 04:18:13.704432 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:13.704405    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r" podUID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Oct 31 04:18:14.553438 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:14.553406    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:14.709277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:14.709244    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:18:14.710339 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:14.710317    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:18:14.749462 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:14.749438    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:15.734834 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:15.734777    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:17.070781 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.070745    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"]
Oct 31 04:18:17.071363 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.071347    4900 cpu_manager.go:414] "containerMap: removing container" podUID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerName="istio-proxy"
Oct 31 04:18:17.071410 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.071365    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerName="istio-proxy"
Oct 31 04:18:17.071410 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.071383    4900 cpu_manager.go:414] "containerMap: removing container" podUID="70848e1b-1960-4cf9-bf4e-fabd84a3e16d" containerName="payload-pre-processing"
Oct 31 04:18:17.071410 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.071388    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70848e1b-1960-4cf9-bf4e-fabd84a3e16d" containerName="payload-pre-processing"
Oct 31 04:18:17.071410 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.071408    4900 cpu_manager.go:414] "containerMap: removing container" podUID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerName="istio-proxy"
Oct 31 04:18:17.071551 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.071413    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerName="istio-proxy"
Oct 31 04:18:17.071551 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.071501    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerName="istio-proxy"
Oct 31 04:18:17.071551 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.071511    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="70848e1b-1960-4cf9-bf4e-fabd84a3e16d" containerName="payload-pre-processing"
Oct 31 04:18:17.071551 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.071519    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="7af499b8-dbd8-4a13-b009-af4aff66fc7b" containerName="istio-proxy"
Oct 31 04:18:17.073966 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.073948    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.166842 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.166783    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.167013 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.166889    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.167013 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.166926    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.167130 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.167013    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.167192 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.167156    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvq7h\" (UniqueName: \"kubernetes.io/projected/9d97d1f9-e5a1-4768-9940-f81b31df1f44-kube-api-access-qvq7h\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.167254 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.167224    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.167310 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.167297    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.167467 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.167452    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.167550 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.167526    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.182239 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.182203    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"]
Oct 31 04:18:17.276665 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.276613    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.276892 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.276678    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.276892 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.276718    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.276892 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.276757    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.276892 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.276805    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.277121 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.276911    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvq7h\" (UniqueName: \"kubernetes.io/projected/9d97d1f9-e5a1-4768-9940-f81b31df1f44-kube-api-access-qvq7h\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.277121 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.276959    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.277121 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.277010    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.277281 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.277168    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.280540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.280512    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.281090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.281068    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.286498 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.283930    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.286498 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.284189    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.286498 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.284414    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.286498 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.285325    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.286498 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.285332    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.338520 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.338428    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.341769 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.341733    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvq7h\" (UniqueName: \"kubernetes.io/projected/9d97d1f9-e5a1-4768-9940-f81b31df1f44-kube-api-access-qvq7h\") pod \"maas-default-gateway-openshift-default-6886b96f9-gl6cz\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.386322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.386287    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:17.433164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.433126    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:17.813562 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:18:17.813532    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d97d1f9_e5a1_4768_9940_f81b31df1f44.slice/crio-10185bcb221a22a039951c0dada3709ad637f94848e0a08c5c3ea11d8144315b WatchSource:0}: Error finding container 10185bcb221a22a039951c0dada3709ad637f94848e0a08c5c3ea11d8144315b: Status 404 returned error can't find the container with id 10185bcb221a22a039951c0dada3709ad637f94848e0a08c5c3ea11d8144315b
Oct 31 04:18:17.815369 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.815339    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"]
Oct 31 04:18:17.816112 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.816081    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:18:17.816161 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.816149    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:18:17.816209 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:17.816192    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:18:18.382915 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:18.382861    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz" event={"ID":"9d97d1f9-e5a1-4768-9940-f81b31df1f44","Type":"ContainerStarted","Data":"7f32167da6fef0eb1065ae58aad9615f0956ad43f4f35b860bcce4051860227d"}
Oct 31 04:18:18.383619 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:18.383587    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz" event={"ID":"9d97d1f9-e5a1-4768-9940-f81b31df1f44","Type":"ContainerStarted","Data":"10185bcb221a22a039951c0dada3709ad637f94848e0a08c5c3ea11d8144315b"}
Oct 31 04:18:18.389949 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:18.389911    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:18.392561 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:18.392524    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gl6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 31 04:18:18.392725 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:18.392592    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz" podUID="9d97d1f9-e5a1-4768-9940-f81b31df1f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 31 04:18:18.513083 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:18.513031    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz" podStartSLOduration=1.513014488 podStartE2EDuration="1.513014488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:18.51043419 +0000 UTC m=+4251.476767534" watchObservedRunningTime="2026-10-31 04:18:18.513014488 +0000 UTC m=+4251.479347833"
Oct 31 04:18:19.387695 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:19.387663    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gl6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 31 04:18:19.388125 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:19.387714    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz" podUID="9d97d1f9-e5a1-4768-9940-f81b31df1f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 31 04:18:20.387647 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:20.387609    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gl6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 31 04:18:20.387852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:20.387663    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz" podUID="9d97d1f9-e5a1-4768-9940-f81b31df1f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 31 04:18:21.290655 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:21.290586    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:21.387288 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:21.387256    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gl6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 31 04:18:21.387470 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:21.387310    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz" podUID="9d97d1f9-e5a1-4768-9940-f81b31df1f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 31 04:18:22.386869 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:22.386830    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gl6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 31 04:18:22.387273 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:22.386888    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz" podUID="9d97d1f9-e5a1-4768-9940-f81b31df1f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 31 04:18:22.610370 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:22.610335    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:23.387283 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:23.387247    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gl6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 31 04:18:23.387689 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:23.387324    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz" podUID="9d97d1f9-e5a1-4768-9940-f81b31df1f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 31 04:18:24.387501 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:24.387466    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gl6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 31 04:18:24.387951 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:24.387528    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz" podUID="9d97d1f9-e5a1-4768-9940-f81b31df1f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 31 04:18:25.387610 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:25.387566    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-gl6cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Oct 31 04:18:25.388070 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:25.387640    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz" podUID="9d97d1f9-e5a1-4768-9940-f81b31df1f44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Oct 31 04:18:26.392222 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:26.392184    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:26.392674 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:26.392643    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:26.393415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:26.393394    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:18:27.496211 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:27.496173    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:32.382701 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:32.382668    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:36.983723 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:18:36.983680    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:18:37.331036 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:37.330778    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:42.377746 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:42.377692    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:43.785538 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:43.785504    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:48.649488 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:48.649451    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:48.861140 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:48.861101    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:49.148071 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:49.148012    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:49.923535 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:49.923486    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:50.151110 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:50.151073    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:50.691744 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:50.691709    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:50.908653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:50.908621    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:51.810752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:51.810717    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:52.030385 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:52.030354    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:54.183241 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:54.183211    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:55.677181 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:55.677152    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:56.653693 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:56.653655    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:18:59.435335 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:18:59.435300    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:04.413079 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:04.413046    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:05.831052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:05.831023    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:06.560069 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:06.560032    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:06.769630 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:06.769595    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:06.985108 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:19:06.985072    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:19:09.297160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:09.297062    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:11.873284 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:11.873252    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:12.648202 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:12.648165    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:13.586359 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:13.586328    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:15.404018 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:15.403982    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:18.282605 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:18.282560    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:18.502763 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:18.502721    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:19.891187 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:19.891158    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:20.338999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:20.338956    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"]
Oct 31 04:19:20.339266 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:20.339239    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px" podUID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerName="istio-proxy" containerID="cri-o://de856a78d19581d7d2d9360123956461bb7ea74589041e60fb8a60c364ce0b14" gracePeriod=30
Oct 31 04:19:20.380977 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:20.380947    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:20.583850 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:20.583783    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:21.475639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:21.475604    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:21.692674 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:21.692634    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:22.727597 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:22.727563    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:22.944737 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:22.944686    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:25.024517 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.024484    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"]
Oct 31 04:19:25.025030 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.024695    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k" podUID="2e5fe713-f05f-4052-8621-ac82812d0000" containerName="maas-api" containerID="cri-o://20b938ac97bbc2d6a82dd41fddfd49a75b7de8a5afb0b1b219b9c69dfafcedc1" gracePeriod=30
Oct 31 04:19:25.231132 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.231097    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:25.301175 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.301147    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:19:25.380015 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.379977    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e5fe713-f05f-4052-8621-ac82812d0000-kube-api-access-8fpcz\" (UniqueName: \"kubernetes.io/projected/2e5fe713-f05f-4052-8621-ac82812d0000-kube-api-access-8fpcz\") pod \"2e5fe713-f05f-4052-8621-ac82812d0000\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") "
Oct 31 04:19:25.380156 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.380058    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e5fe713-f05f-4052-8621-ac82812d0000-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5fe713-f05f-4052-8621-ac82812d0000-openshift-service-ca\") pod \"2e5fe713-f05f-4052-8621-ac82812d0000\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") "
Oct 31 04:19:25.380156 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.380120    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-maas-api-tls\") pod \"2e5fe713-f05f-4052-8621-ac82812d0000\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") "
Oct 31 04:19:25.380240 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.380168    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-metrics-certs\") pod \"2e5fe713-f05f-4052-8621-ac82812d0000\" (UID: \"2e5fe713-f05f-4052-8621-ac82812d0000\") "
Oct 31 04:19:25.380472 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.380444    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e5fe713-f05f-4052-8621-ac82812d0000-openshift-service-ca" pod "2e5fe713-f05f-4052-8621-ac82812d0000" (UID: "2e5fe713-f05f-4052-8621-ac82812d0000"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:19:25.380811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.380760    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e5fe713-f05f-4052-8621-ac82812d0000-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:25.382224 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.382197    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-maas-api-tls" pod "2e5fe713-f05f-4052-8621-ac82812d0000" (UID: "2e5fe713-f05f-4052-8621-ac82812d0000"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:19:25.382329 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.382244    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-metrics-certs" pod "2e5fe713-f05f-4052-8621-ac82812d0000" (UID: "2e5fe713-f05f-4052-8621-ac82812d0000"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:19:25.382329 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.382290    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e5fe713-f05f-4052-8621-ac82812d0000-kube-api-access-8fpcz" pod "2e5fe713-f05f-4052-8621-ac82812d0000" (UID: "2e5fe713-f05f-4052-8621-ac82812d0000"). InnerVolumeSpecName "kube-api-access-8fpcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:19:25.482737 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.482696    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fpcz\" (UniqueName: \"kubernetes.io/projected/2e5fe713-f05f-4052-8621-ac82812d0000-kube-api-access-8fpcz\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:25.482737 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.482738    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:25.483021 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.482754    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e5fe713-f05f-4052-8621-ac82812d0000-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:25.616453 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.616425    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:19:25.686712 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.686677    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-token\" (UniqueName: \"kubernetes.io/projected/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-token\") pod \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") "
Oct 31 04:19:25.686937 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.686757    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-workload-certs\") pod \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") "
Oct 31 04:19:25.686937 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.686802    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istiod-ca-cert\") pod \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") "
Oct 31 04:19:25.686937 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.686837    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-data\") pod \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") "
Oct 31 04:19:25.687117 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.686947    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b94bdc-4427-400c-ad3f-5032aa82a83f-kube-api-access-6zp5w\" (UniqueName: \"kubernetes.io/projected/e6b94bdc-4427-400c-ad3f-5032aa82a83f-kube-api-access-6zp5w\") pod \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") "
Oct 31 04:19:25.687117 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.687050    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-podinfo\") pod \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") "
Oct 31 04:19:25.687117 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.687099    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-credential-socket\") pod \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") "
Oct 31 04:19:25.687276 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.687112    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-workload-certs" pod "e6b94bdc-4427-400c-ad3f-5032aa82a83f" (UID: "e6b94bdc-4427-400c-ad3f-5032aa82a83f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:19:25.687276 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.687132    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-workload-socket\") pod \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") "
Oct 31 04:19:25.687276 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.687158    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-data" pod "e6b94bdc-4427-400c-ad3f-5032aa82a83f" (UID: "e6b94bdc-4427-400c-ad3f-5032aa82a83f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:19:25.687276 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.687234    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istiod-ca-cert" pod "e6b94bdc-4427-400c-ad3f-5032aa82a83f" (UID: "e6b94bdc-4427-400c-ad3f-5032aa82a83f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:19:25.687276 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.687264    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-envoy\") pod \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\" (UID: \"e6b94bdc-4427-400c-ad3f-5032aa82a83f\") "
Oct 31 04:19:25.687524 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.687380    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-credential-socket" pod "e6b94bdc-4427-400c-ad3f-5032aa82a83f" (UID: "e6b94bdc-4427-400c-ad3f-5032aa82a83f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:19:25.687524 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.687490    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-workload-socket" pod "e6b94bdc-4427-400c-ad3f-5032aa82a83f" (UID: "e6b94bdc-4427-400c-ad3f-5032aa82a83f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:19:25.687918 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.687891    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:25.688052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.687921    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:25.688052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.687937    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:25.688052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.687953    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:25.688052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.687967    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:25.689157 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.689133    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-token" pod "e6b94bdc-4427-400c-ad3f-5032aa82a83f" (UID: "e6b94bdc-4427-400c-ad3f-5032aa82a83f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:19:25.689288 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.689268    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b94bdc-4427-400c-ad3f-5032aa82a83f-kube-api-access-6zp5w" pod "e6b94bdc-4427-400c-ad3f-5032aa82a83f" (UID: "e6b94bdc-4427-400c-ad3f-5032aa82a83f"). InnerVolumeSpecName "kube-api-access-6zp5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:19:25.689474 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.689453    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-podinfo" pod "e6b94bdc-4427-400c-ad3f-5032aa82a83f" (UID: "e6b94bdc-4427-400c-ad3f-5032aa82a83f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:19:25.689540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.689468    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-envoy" pod "e6b94bdc-4427-400c-ad3f-5032aa82a83f" (UID: "e6b94bdc-4427-400c-ad3f-5032aa82a83f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:19:25.789382 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.789291    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:25.789382 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.789323    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:25.789382 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.789334    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zp5w\" (UniqueName: \"kubernetes.io/projected/e6b94bdc-4427-400c-ad3f-5032aa82a83f-kube-api-access-6zp5w\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:25.789382 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.789346    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b94bdc-4427-400c-ad3f-5032aa82a83f-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:25.795873 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.795839    4900 generic.go:356] "Generic (PLEG): container finished" podID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerID="de856a78d19581d7d2d9360123956461bb7ea74589041e60fb8a60c364ce0b14" exitCode=0
Oct 31 04:19:25.796037 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.795898    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"
Oct 31 04:19:25.796037 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.795921    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px" event={"ID":"e6b94bdc-4427-400c-ad3f-5032aa82a83f","Type":"ContainerDied","Data":"de856a78d19581d7d2d9360123956461bb7ea74589041e60fb8a60c364ce0b14"}
Oct 31 04:19:25.796037 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.796012    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px" event={"ID":"e6b94bdc-4427-400c-ad3f-5032aa82a83f","Type":"ContainerDied","Data":"8860a929de4e462255c830f2caacd701bb13c090cdbf2cd2722b02edfa9216e8"}
Oct 31 04:19:25.796212 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.796042    4900 scope.go:122] "RemoveContainer" containerID="de856a78d19581d7d2d9360123956461bb7ea74589041e60fb8a60c364ce0b14"
Oct 31 04:19:25.797555 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.797525    4900 generic.go:356] "Generic (PLEG): container finished" podID="2e5fe713-f05f-4052-8621-ac82812d0000" containerID="20b938ac97bbc2d6a82dd41fddfd49a75b7de8a5afb0b1b219b9c69dfafcedc1" exitCode=0
Oct 31 04:19:25.797697 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.797566    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k" event={"ID":"2e5fe713-f05f-4052-8621-ac82812d0000","Type":"ContainerDied","Data":"20b938ac97bbc2d6a82dd41fddfd49a75b7de8a5afb0b1b219b9c69dfafcedc1"}
Oct 31 04:19:25.797697 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.797601    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k" event={"ID":"2e5fe713-f05f-4052-8621-ac82812d0000","Type":"ContainerDied","Data":"9f2ce87fb10b5e6b706064c159e374d59a2674c24c75ffead5910d0f1e8dd2bc"}
Oct 31 04:19:25.797697 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.797600    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"
Oct 31 04:19:25.806299 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.806275    4900 scope.go:122] "RemoveContainer" containerID="e8419a3d511e712c305033d9a5f013a54b9a48c9d956ede40006ff61fcaf2fce"
Oct 31 04:19:25.815543 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.815526    4900 scope.go:122] "RemoveContainer" containerID="de856a78d19581d7d2d9360123956461bb7ea74589041e60fb8a60c364ce0b14"
Oct 31 04:19:25.815846 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:19:25.815771    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de856a78d19581d7d2d9360123956461bb7ea74589041e60fb8a60c364ce0b14\": container with ID starting with de856a78d19581d7d2d9360123956461bb7ea74589041e60fb8a60c364ce0b14 not found: ID does not exist" containerID="de856a78d19581d7d2d9360123956461bb7ea74589041e60fb8a60c364ce0b14"
Oct 31 04:19:25.815942 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.815854    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de856a78d19581d7d2d9360123956461bb7ea74589041e60fb8a60c364ce0b14"} err="failed to get container status \"de856a78d19581d7d2d9360123956461bb7ea74589041e60fb8a60c364ce0b14\": rpc error: code = NotFound desc = could not find container \"de856a78d19581d7d2d9360123956461bb7ea74589041e60fb8a60c364ce0b14\": container with ID starting with de856a78d19581d7d2d9360123956461bb7ea74589041e60fb8a60c364ce0b14 not found: ID does not exist"
Oct 31 04:19:25.815942 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.815873    4900 scope.go:122] "RemoveContainer" containerID="e8419a3d511e712c305033d9a5f013a54b9a48c9d956ede40006ff61fcaf2fce"
Oct 31 04:19:25.816137 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:19:25.816119    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8419a3d511e712c305033d9a5f013a54b9a48c9d956ede40006ff61fcaf2fce\": container with ID starting with e8419a3d511e712c305033d9a5f013a54b9a48c9d956ede40006ff61fcaf2fce not found: ID does not exist" containerID="e8419a3d511e712c305033d9a5f013a54b9a48c9d956ede40006ff61fcaf2fce"
Oct 31 04:19:25.816181 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.816144    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8419a3d511e712c305033d9a5f013a54b9a48c9d956ede40006ff61fcaf2fce"} err="failed to get container status \"e8419a3d511e712c305033d9a5f013a54b9a48c9d956ede40006ff61fcaf2fce\": rpc error: code = NotFound desc = could not find container \"e8419a3d511e712c305033d9a5f013a54b9a48c9d956ede40006ff61fcaf2fce\": container with ID starting with e8419a3d511e712c305033d9a5f013a54b9a48c9d956ede40006ff61fcaf2fce not found: ID does not exist"
Oct 31 04:19:25.816181 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.816160    4900 scope.go:122] "RemoveContainer" containerID="20b938ac97bbc2d6a82dd41fddfd49a75b7de8a5afb0b1b219b9c69dfafcedc1"
Oct 31 04:19:25.825141 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.825123    4900 scope.go:122] "RemoveContainer" containerID="20b938ac97bbc2d6a82dd41fddfd49a75b7de8a5afb0b1b219b9c69dfafcedc1"
Oct 31 04:19:25.825413 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:19:25.825394    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b938ac97bbc2d6a82dd41fddfd49a75b7de8a5afb0b1b219b9c69dfafcedc1\": container with ID starting with 20b938ac97bbc2d6a82dd41fddfd49a75b7de8a5afb0b1b219b9c69dfafcedc1 not found: ID does not exist" containerID="20b938ac97bbc2d6a82dd41fddfd49a75b7de8a5afb0b1b219b9c69dfafcedc1"
Oct 31 04:19:25.825466 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.825422    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b938ac97bbc2d6a82dd41fddfd49a75b7de8a5afb0b1b219b9c69dfafcedc1"} err="failed to get container status \"20b938ac97bbc2d6a82dd41fddfd49a75b7de8a5afb0b1b219b9c69dfafcedc1\": rpc error: code = NotFound desc = could not find container \"20b938ac97bbc2d6a82dd41fddfd49a75b7de8a5afb0b1b219b9c69dfafcedc1\": container with ID starting with 20b938ac97bbc2d6a82dd41fddfd49a75b7de8a5afb0b1b219b9c69dfafcedc1 not found: ID does not exist"
Oct 31 04:19:25.960584 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.960528    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"]
Oct 31 04:19:25.963963 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:25.963929    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e679fe-5c6bf75479-chw7k"]
Oct 31 04:19:26.073115 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:26.073080    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"]
Oct 31 04:19:26.077275 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:26.077248    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6f30f2-openshift-default-94c897f67-n54px"]
Oct 31 04:19:26.986287 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:26.986253    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:27.153684 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:27.153647    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e5fe713-f05f-4052-8621-ac82812d0000" path="/var/lib/kubelet/pods/2e5fe713-f05f-4052-8621-ac82812d0000/volumes"
Oct 31 04:19:27.154203 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:27.154187    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" path="/var/lib/kubelet/pods/e6b94bdc-4427-400c-ad3f-5032aa82a83f/volumes"
Oct 31 04:19:28.917268 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:28.917238    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:29.140574 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:29.140538    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:30.486242 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:30.486210    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:30.936175 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:30.936090    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"]
Oct 31 04:19:30.936343 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:30.936308    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq" podUID="c0e37b79-c673-4bb9-a740-eda25789449a" containerName="istio-proxy" containerID="cri-o://a72b1b27134fe96ac1bf6789a7cee605154a8cb6b4e925017e52938ebc70baa5" gracePeriod=30
Oct 31 04:19:31.016417 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:31.016381    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:32.006218 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:32.006183    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:32.243922 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:32.243883    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:32.460425 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:32.460273    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:35.624864 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:35.624823    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:36.216256 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.216193    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:19:36.247303 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.247269    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-workload-certs\") pod \"c0e37b79-c673-4bb9-a740-eda25789449a\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") "
Oct 31 04:19:36.247458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.247326    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-credential-socket\") pod \"c0e37b79-c673-4bb9-a740-eda25789449a\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") "
Oct 31 04:19:36.247458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.247366    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-workload-socket\") pod \"c0e37b79-c673-4bb9-a740-eda25789449a\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") "
Oct 31 04:19:36.247555 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.247482    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e37b79-c673-4bb9-a740-eda25789449a-istio-token\" (UniqueName: \"kubernetes.io/projected/c0e37b79-c673-4bb9-a740-eda25789449a-istio-token\") pod \"c0e37b79-c673-4bb9-a740-eda25789449a\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") "
Oct 31 04:19:36.247555 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.247522    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-istio-envoy\") pod \"c0e37b79-c673-4bb9-a740-eda25789449a\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") "
Oct 31 04:19:36.247675 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.247546    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-workload-certs" pod "c0e37b79-c673-4bb9-a740-eda25789449a" (UID: "c0e37b79-c673-4bb9-a740-eda25789449a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:19:36.247675 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.247581    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-credential-socket" pod "c0e37b79-c673-4bb9-a740-eda25789449a" (UID: "c0e37b79-c673-4bb9-a740-eda25789449a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:19:36.247675 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.247641    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e37b79-c673-4bb9-a740-eda25789449a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e37b79-c673-4bb9-a740-eda25789449a-istiod-ca-cert\") pod \"c0e37b79-c673-4bb9-a740-eda25789449a\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") "
Oct 31 04:19:36.247675 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.247648    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-workload-socket" pod "c0e37b79-c673-4bb9-a740-eda25789449a" (UID: "c0e37b79-c673-4bb9-a740-eda25789449a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:19:36.247931 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.247684    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-istio-data\") pod \"c0e37b79-c673-4bb9-a740-eda25789449a\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") "
Oct 31 04:19:36.247931 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.247737    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e37b79-c673-4bb9-a740-eda25789449a-kube-api-access-gzjh6\" (UniqueName: \"kubernetes.io/projected/c0e37b79-c673-4bb9-a740-eda25789449a-kube-api-access-gzjh6\") pod \"c0e37b79-c673-4bb9-a740-eda25789449a\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") "
Oct 31 04:19:36.247931 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.247814    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0e37b79-c673-4bb9-a740-eda25789449a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e37b79-c673-4bb9-a740-eda25789449a-istio-podinfo\") pod \"c0e37b79-c673-4bb9-a740-eda25789449a\" (UID: \"c0e37b79-c673-4bb9-a740-eda25789449a\") "
Oct 31 04:19:36.248108 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.247967    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-istio-data" pod "c0e37b79-c673-4bb9-a740-eda25789449a" (UID: "c0e37b79-c673-4bb9-a740-eda25789449a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:19:36.248108 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.248063    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e37b79-c673-4bb9-a740-eda25789449a-istiod-ca-cert" pod "c0e37b79-c673-4bb9-a740-eda25789449a" (UID: "c0e37b79-c673-4bb9-a740-eda25789449a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:19:36.248415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.248392    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:36.248500 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.248420    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:36.248500 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.248436    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:36.248500 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.248450    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0e37b79-c673-4bb9-a740-eda25789449a-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:36.248500 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.248463    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:36.249873 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.249847    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e37b79-c673-4bb9-a740-eda25789449a-istio-token" pod "c0e37b79-c673-4bb9-a740-eda25789449a" (UID: "c0e37b79-c673-4bb9-a740-eda25789449a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:19:36.249995 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.249933    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0e37b79-c673-4bb9-a740-eda25789449a-istio-podinfo" pod "c0e37b79-c673-4bb9-a740-eda25789449a" (UID: "c0e37b79-c673-4bb9-a740-eda25789449a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:19:36.250142 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.250126    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e37b79-c673-4bb9-a740-eda25789449a-kube-api-access-gzjh6" pod "c0e37b79-c673-4bb9-a740-eda25789449a" (UID: "c0e37b79-c673-4bb9-a740-eda25789449a"). InnerVolumeSpecName "kube-api-access-gzjh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:19:36.250142 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.250131    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-istio-envoy" pod "c0e37b79-c673-4bb9-a740-eda25789449a" (UID: "c0e37b79-c673-4bb9-a740-eda25789449a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:19:36.349502 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.349468    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzjh6\" (UniqueName: \"kubernetes.io/projected/c0e37b79-c673-4bb9-a740-eda25789449a-kube-api-access-gzjh6\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:36.349502 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.349495    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0e37b79-c673-4bb9-a740-eda25789449a-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:36.349502 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.349506    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0e37b79-c673-4bb9-a740-eda25789449a-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:36.349746 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.349515    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0e37b79-c673-4bb9-a740-eda25789449a-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:36.890986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.890951    4900 generic.go:356] "Generic (PLEG): container finished" podID="c0e37b79-c673-4bb9-a740-eda25789449a" containerID="a72b1b27134fe96ac1bf6789a7cee605154a8cb6b4e925017e52938ebc70baa5" exitCode=0
Oct 31 04:19:36.891527 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.891001    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"
Oct 31 04:19:36.891527 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.891025    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq" event={"ID":"c0e37b79-c673-4bb9-a740-eda25789449a","Type":"ContainerDied","Data":"a72b1b27134fe96ac1bf6789a7cee605154a8cb6b4e925017e52938ebc70baa5"}
Oct 31 04:19:36.891527 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.891076    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq" event={"ID":"c0e37b79-c673-4bb9-a740-eda25789449a","Type":"ContainerDied","Data":"55637c7d8731b8e9c810831ffb0f5308ffd14d3788b3a370604b2e79eece7543"}
Oct 31 04:19:36.891527 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.891105    4900 scope.go:122] "RemoveContainer" containerID="a72b1b27134fe96ac1bf6789a7cee605154a8cb6b4e925017e52938ebc70baa5"
Oct 31 04:19:36.901475 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.901459    4900 scope.go:122] "RemoveContainer" containerID="569f4ec56e74a1e590dc0b2b2981dccbaf482b9ecb0d49aff3ea33e330735a10"
Oct 31 04:19:36.910260 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.910236    4900 scope.go:122] "RemoveContainer" containerID="a72b1b27134fe96ac1bf6789a7cee605154a8cb6b4e925017e52938ebc70baa5"
Oct 31 04:19:36.910558 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:19:36.910539    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72b1b27134fe96ac1bf6789a7cee605154a8cb6b4e925017e52938ebc70baa5\": container with ID starting with a72b1b27134fe96ac1bf6789a7cee605154a8cb6b4e925017e52938ebc70baa5 not found: ID does not exist" containerID="a72b1b27134fe96ac1bf6789a7cee605154a8cb6b4e925017e52938ebc70baa5"
Oct 31 04:19:36.910647 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.910572    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72b1b27134fe96ac1bf6789a7cee605154a8cb6b4e925017e52938ebc70baa5"} err="failed to get container status \"a72b1b27134fe96ac1bf6789a7cee605154a8cb6b4e925017e52938ebc70baa5\": rpc error: code = NotFound desc = could not find container \"a72b1b27134fe96ac1bf6789a7cee605154a8cb6b4e925017e52938ebc70baa5\": container with ID starting with a72b1b27134fe96ac1bf6789a7cee605154a8cb6b4e925017e52938ebc70baa5 not found: ID does not exist"
Oct 31 04:19:36.910647 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.910597    4900 scope.go:122] "RemoveContainer" containerID="569f4ec56e74a1e590dc0b2b2981dccbaf482b9ecb0d49aff3ea33e330735a10"
Oct 31 04:19:36.910892 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:19:36.910875    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"569f4ec56e74a1e590dc0b2b2981dccbaf482b9ecb0d49aff3ea33e330735a10\": container with ID starting with 569f4ec56e74a1e590dc0b2b2981dccbaf482b9ecb0d49aff3ea33e330735a10 not found: ID does not exist" containerID="569f4ec56e74a1e590dc0b2b2981dccbaf482b9ecb0d49aff3ea33e330735a10"
Oct 31 04:19:36.910957 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:36.910900    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"569f4ec56e74a1e590dc0b2b2981dccbaf482b9ecb0d49aff3ea33e330735a10"} err="failed to get container status \"569f4ec56e74a1e590dc0b2b2981dccbaf482b9ecb0d49aff3ea33e330735a10\": rpc error: code = NotFound desc = could not find container \"569f4ec56e74a1e590dc0b2b2981dccbaf482b9ecb0d49aff3ea33e330735a10\": container with ID starting with 569f4ec56e74a1e590dc0b2b2981dccbaf482b9ecb0d49aff3ea33e330735a10 not found: ID does not exist"
Oct 31 04:19:36.984129 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:19:36.984094    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:19:37.055514 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:37.055486    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"]
Oct 31 04:19:37.062150 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:37.062122    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e679fe-openshift-default-774df5789c-tmhrq"]
Oct 31 04:19:37.153450 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:37.153370    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e37b79-c673-4bb9-a740-eda25789449a" path="/var/lib/kubelet/pods/c0e37b79-c673-4bb9-a740-eda25789449a/volumes"
Oct 31 04:19:37.473142 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:37.473062    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:38.402286 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:38.402251    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:38.643425 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:38.643394    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:38.999503 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:38.999467    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:41.715477 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:41.715438    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:42.391108 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:42.391076    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:45.050814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:45.050765    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:45.253370 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:45.253333    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:46.498392 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:46.498354    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"]
Oct 31 04:19:46.688967 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:46.688935    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:46.950436 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:46.950327    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c" podUID="3e59d7a0-72ed-4065-9538-cc15b531b5fe" containerName="maas-api" containerID="cri-o://6855f932a2ceedcdcaa9bda287c74bfad763b5df2e07a22115aeb99def30d8dd" gracePeriod=30
Oct 31 04:19:47.221596 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.221567    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:19:47.389619 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.389585    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e59d7a0-72ed-4065-9538-cc15b531b5fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e59d7a0-72ed-4065-9538-cc15b531b5fe-openshift-service-ca\") pod \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") "
Oct 31 04:19:47.389860 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.389661    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-metrics-certs\") pod \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") "
Oct 31 04:19:47.389860 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.389761    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-maas-api-tls\") pod \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") "
Oct 31 04:19:47.389860 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.389824    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e59d7a0-72ed-4065-9538-cc15b531b5fe-kube-api-access-bb9xs\" (UniqueName: \"kubernetes.io/projected/3e59d7a0-72ed-4065-9538-cc15b531b5fe-kube-api-access-bb9xs\") pod \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\" (UID: \"3e59d7a0-72ed-4065-9538-cc15b531b5fe\") "
Oct 31 04:19:47.390098 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.390075    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e59d7a0-72ed-4065-9538-cc15b531b5fe-openshift-service-ca" pod "3e59d7a0-72ed-4065-9538-cc15b531b5fe" (UID: "3e59d7a0-72ed-4065-9538-cc15b531b5fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:19:47.390399 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.390374    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e59d7a0-72ed-4065-9538-cc15b531b5fe-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:47.391890 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.391860    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-metrics-certs" pod "3e59d7a0-72ed-4065-9538-cc15b531b5fe" (UID: "3e59d7a0-72ed-4065-9538-cc15b531b5fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:19:47.392000 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.391931    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-maas-api-tls" pod "3e59d7a0-72ed-4065-9538-cc15b531b5fe" (UID: "3e59d7a0-72ed-4065-9538-cc15b531b5fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:19:47.392067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.392032    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e59d7a0-72ed-4065-9538-cc15b531b5fe-kube-api-access-bb9xs" pod "3e59d7a0-72ed-4065-9538-cc15b531b5fe" (UID: "3e59d7a0-72ed-4065-9538-cc15b531b5fe"). InnerVolumeSpecName "kube-api-access-bb9xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:19:47.505835 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.505799    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:47.506398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.506376    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e59d7a0-72ed-4065-9538-cc15b531b5fe-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:47.506510 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.506497    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb9xs\" (UniqueName: \"kubernetes.io/projected/3e59d7a0-72ed-4065-9538-cc15b531b5fe-kube-api-access-bb9xs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:19:47.957018 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.956930    4900 generic.go:356] "Generic (PLEG): container finished" podID="3e59d7a0-72ed-4065-9538-cc15b531b5fe" containerID="6855f932a2ceedcdcaa9bda287c74bfad763b5df2e07a22115aeb99def30d8dd" exitCode=0
Oct 31 04:19:47.957018 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.956995    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"
Oct 31 04:19:47.957212 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.957018    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c" event={"ID":"3e59d7a0-72ed-4065-9538-cc15b531b5fe","Type":"ContainerDied","Data":"6855f932a2ceedcdcaa9bda287c74bfad763b5df2e07a22115aeb99def30d8dd"}
Oct 31 04:19:47.957212 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.957079    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c" event={"ID":"3e59d7a0-72ed-4065-9538-cc15b531b5fe","Type":"ContainerDied","Data":"ff1aff81cda83cc412dadefb4a3bbad1f7c0f315793cb6fc3a6dab8f08f2e8f3"}
Oct 31 04:19:47.957212 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.957109    4900 scope.go:122] "RemoveContainer" containerID="6855f932a2ceedcdcaa9bda287c74bfad763b5df2e07a22115aeb99def30d8dd"
Oct 31 04:19:47.978523 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.978495    4900 scope.go:122] "RemoveContainer" containerID="6855f932a2ceedcdcaa9bda287c74bfad763b5df2e07a22115aeb99def30d8dd"
Oct 31 04:19:47.978895 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:19:47.978865    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6855f932a2ceedcdcaa9bda287c74bfad763b5df2e07a22115aeb99def30d8dd\": container with ID starting with 6855f932a2ceedcdcaa9bda287c74bfad763b5df2e07a22115aeb99def30d8dd not found: ID does not exist" containerID="6855f932a2ceedcdcaa9bda287c74bfad763b5df2e07a22115aeb99def30d8dd"
Oct 31 04:19:47.978990 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:47.978896    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6855f932a2ceedcdcaa9bda287c74bfad763b5df2e07a22115aeb99def30d8dd"} err="failed to get container status \"6855f932a2ceedcdcaa9bda287c74bfad763b5df2e07a22115aeb99def30d8dd\": rpc error: code = NotFound desc = could not find container \"6855f932a2ceedcdcaa9bda287c74bfad763b5df2e07a22115aeb99def30d8dd\": container with ID starting with 6855f932a2ceedcdcaa9bda287c74bfad763b5df2e07a22115aeb99def30d8dd not found: ID does not exist"
Oct 31 04:19:48.140018 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:48.139978    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"]
Oct 31 04:19:48.189554 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:48.189520    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c42c43-555748c86c-8dd6c"]
Oct 31 04:19:48.507146 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:48.507112    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:49.153139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:49.153105    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e59d7a0-72ed-4065-9538-cc15b531b5fe" path="/var/lib/kubelet/pods/3e59d7a0-72ed-4065-9538-cc15b531b5fe/volumes"
Oct 31 04:19:53.344305 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:53.344233    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:54.754334 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:54.754295    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:54.993466 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:54.993434    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:56.352583 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:56.352553    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:56.798589 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:56.798561    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"]
Oct 31 04:19:56.798809 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:56.798765    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r" podUID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerName="istio-proxy" containerID="cri-o://ccccaebf035067f2154410fd2216959e1d515fae4c6d59a3563251ac02c0b969" gracePeriod=30
Oct 31 04:19:56.884297 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:56.884261    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:58.559012 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:58.558977    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:59.058694 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:59.058656    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:19:59.712006 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:59.711942    4900 patch_prober.go:29] interesting pod/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:19:59.712684 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:19:59.712642    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r" podUID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:20:02.164963 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.164653    4900 generic.go:356] "Generic (PLEG): container finished" podID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerID="ccccaebf035067f2154410fd2216959e1d515fae4c6d59a3563251ac02c0b969" exitCode=0
Oct 31 04:20:02.164963 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.164749    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r" event={"ID":"6ed827b7-7ac6-412d-ade0-70ce0d8cc399","Type":"ContainerDied","Data":"ccccaebf035067f2154410fd2216959e1d515fae4c6d59a3563251ac02c0b969"}
Oct 31 04:20:02.164963 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.164825    4900 scope.go:122] "RemoveContainer" containerID="a096ad3d8e428037aaf1a67313426834d9b7411c2a2119c8db836703ff6f879b"
Oct 31 04:20:02.230709 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.230676    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:20:02.328578 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.328538    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-workload-certs\") pod \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") "
Oct 31 04:20:02.328811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.328608    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-envoy\") pod \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") "
Oct 31 04:20:02.328811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.328660    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-data\") pod \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") "
Oct 31 04:20:02.328811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.328726    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istiod-ca-cert\") pod \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") "
Oct 31 04:20:02.329008 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.328830    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-kube-api-access-4zxwj\" (UniqueName: \"kubernetes.io/projected/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-kube-api-access-4zxwj\") pod \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") "
Oct 31 04:20:02.329008 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.328899    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-token\" (UniqueName: \"kubernetes.io/projected/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-token\") pod \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") "
Oct 31 04:20:02.329008 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.328940    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-credential-socket\") pod \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") "
Oct 31 04:20:02.329201 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.329006    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-workload-certs" pod "6ed827b7-7ac6-412d-ade0-70ce0d8cc399" (UID: "6ed827b7-7ac6-412d-ade0-70ce0d8cc399"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:02.329201 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.329060    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-workload-socket\") pod \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") "
Oct 31 04:20:02.329201 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.329092    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-podinfo\") pod \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\" (UID: \"6ed827b7-7ac6-412d-ade0-70ce0d8cc399\") "
Oct 31 04:20:02.329370 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.329329    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istiod-ca-cert" pod "6ed827b7-7ac6-412d-ade0-70ce0d8cc399" (UID: "6ed827b7-7ac6-412d-ade0-70ce0d8cc399"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:20:02.329861 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.329838    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:02.330033 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.330015    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:02.330664 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.330608    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-workload-socket" pod "6ed827b7-7ac6-412d-ade0-70ce0d8cc399" (UID: "6ed827b7-7ac6-412d-ade0-70ce0d8cc399"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:02.331353 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.330846    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-data" pod "6ed827b7-7ac6-412d-ade0-70ce0d8cc399" (UID: "6ed827b7-7ac6-412d-ade0-70ce0d8cc399"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:02.331353 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.331281    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-credential-socket" pod "6ed827b7-7ac6-412d-ade0-70ce0d8cc399" (UID: "6ed827b7-7ac6-412d-ade0-70ce0d8cc399"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:02.332143 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.332115    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-kube-api-access-4zxwj" pod "6ed827b7-7ac6-412d-ade0-70ce0d8cc399" (UID: "6ed827b7-7ac6-412d-ade0-70ce0d8cc399"). InnerVolumeSpecName "kube-api-access-4zxwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:20:02.333742 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.333715    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-token" pod "6ed827b7-7ac6-412d-ade0-70ce0d8cc399" (UID: "6ed827b7-7ac6-412d-ade0-70ce0d8cc399"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:20:02.334203 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.334182    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-podinfo" pod "6ed827b7-7ac6-412d-ade0-70ce0d8cc399" (UID: "6ed827b7-7ac6-412d-ade0-70ce0d8cc399"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:20:02.342199 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.342167    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-envoy" pod "6ed827b7-7ac6-412d-ade0-70ce0d8cc399" (UID: "6ed827b7-7ac6-412d-ade0-70ce0d8cc399"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:02.431239 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.431205    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:02.431239 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.431233    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:02.431239 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.431244    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:02.431575 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.431255    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:02.431575 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.431268    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:02.431575 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.431276    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:02.431575 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:02.431287    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zxwj\" (UniqueName: \"kubernetes.io/projected/6ed827b7-7ac6-412d-ade0-70ce0d8cc399-kube-api-access-4zxwj\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:03.171995 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:03.171967    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"
Oct 31 04:20:03.172475 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:03.171974    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r" event={"ID":"6ed827b7-7ac6-412d-ade0-70ce0d8cc399","Type":"ContainerDied","Data":"fc9a3b85def1a48dca185c61518da396b0072a25df5468dc65efb5d9a2f5dc33"}
Oct 31 04:20:03.172475 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:03.172103    4900 scope.go:122] "RemoveContainer" containerID="ccccaebf035067f2154410fd2216959e1d515fae4c6d59a3563251ac02c0b969"
Oct 31 04:20:03.390287 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:03.390250    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"]
Oct 31 04:20:03.395463 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:03.395439    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c42c43-openshift-default-64d4f8645-qxn4r"]
Oct 31 04:20:04.069256 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:04.069225    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"]
Oct 31 04:20:04.073762 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:04.073734    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"]
Oct 31 04:20:04.088091 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:04.087833    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"]
Oct 31 04:20:04.189951 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:04.189922    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:04.392681 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:04.392602    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:05.153566 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:05.153534    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" path="/var/lib/kubelet/pods/6ed827b7-7ac6-412d-ade0-70ce0d8cc399/volumes"
Oct 31 04:20:06.273251 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:06.273219    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:06.984959 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:20:06.984921    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:20:07.031897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:07.031864    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:09.196979 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:09.196939    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:12.253055 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:12.253015    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:13.001328 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:13.001283    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:13.226154 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:13.226114    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:14.457996 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:14.457960    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:17.700666 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:17.700631    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:19.093306 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.093263    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main" containerID="cri-o://bbbe1d5e2c75d9d438d0cbc73d754b5ffd92b04905668821494dd4ac3de739b3" gracePeriod=45
Oct 31 04:20:19.094886 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.094858    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d" podUID="5fc487b4-195a-42bb-9db6-4aefa3186c6b" containerName="main" containerID="cri-o://850fe939d0c5f9b08f02333a7f637faa672b007ed8b8464be971a6b4020532ae" gracePeriod=45
Oct 31 04:20:19.108706 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.108670    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp" podUID="e58cb630-6755-44c8-b068-2bbdb6d4fe15" containerName="main" containerID="cri-o://8c72a998c6d8a3e085801d5a2677f767073304a31cc8684e55fda64dfba0b9e9" gracePeriod=45
Oct 31 04:20:19.267309 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.267180    4900 generic.go:356] "Generic (PLEG): container finished" podID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerID="bbbe1d5e2c75d9d438d0cbc73d754b5ffd92b04905668821494dd4ac3de739b3" exitCode=0
Oct 31 04:20:19.267309 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.267257    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" event={"ID":"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63","Type":"ContainerDied","Data":"bbbe1d5e2c75d9d438d0cbc73d754b5ffd92b04905668821494dd4ac3de739b3"}
Oct 31 04:20:19.268660 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.268611    4900 generic.go:356] "Generic (PLEG): container finished" podID="5fc487b4-195a-42bb-9db6-4aefa3186c6b" containerID="850fe939d0c5f9b08f02333a7f637faa672b007ed8b8464be971a6b4020532ae" exitCode=0
Oct 31 04:20:19.268986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.268938    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d" event={"ID":"5fc487b4-195a-42bb-9db6-4aefa3186c6b","Type":"ContainerDied","Data":"850fe939d0c5f9b08f02333a7f637faa672b007ed8b8464be971a6b4020532ae"}
Oct 31 04:20:19.270738 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.270713    4900 generic.go:356] "Generic (PLEG): container finished" podID="e58cb630-6755-44c8-b068-2bbdb6d4fe15" containerID="8c72a998c6d8a3e085801d5a2677f767073304a31cc8684e55fda64dfba0b9e9" exitCode=0
Oct 31 04:20:19.270887 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.270767    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp" event={"ID":"e58cb630-6755-44c8-b068-2bbdb6d4fe15","Type":"ContainerDied","Data":"8c72a998c6d8a3e085801d5a2677f767073304a31cc8684e55fda64dfba0b9e9"}
Oct 31 04:20:19.393108 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.387123    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 04:20:19.415143 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.415114    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 04:20:19.429182 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.429159    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 04:20:19.548423 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548390    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-tmp-dir\") pod \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") "
Oct 31 04:20:19.548603 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548450    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-home\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-home\") pod \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") "
Oct 31 04:20:19.548603 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548480    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc487b4-195a-42bb-9db6-4aefa3186c6b-kube-api-access-jhx5p\" (UniqueName: \"kubernetes.io/projected/5fc487b4-195a-42bb-9db6-4aefa3186c6b-kube-api-access-jhx5p\") pod \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") "
Oct 31 04:20:19.548603 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548554    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fc487b4-195a-42bb-9db6-4aefa3186c6b-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc487b4-195a-42bb-9db6-4aefa3186c6b-tls-certs\") pod \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") "
Oct 31 04:20:19.548603 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548582    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-model-cache\") pod \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") "
Oct 31 04:20:19.548603 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548603    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58cb630-6755-44c8-b068-2bbdb6d4fe15-kube-api-access-s9q5p\" (UniqueName: \"kubernetes.io/projected/e58cb630-6755-44c8-b068-2bbdb6d4fe15-kube-api-access-s9q5p\") pod \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") "
Oct 31 04:20:19.548909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548626    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-kube-api-access-nxqm6\" (UniqueName: \"kubernetes.io/projected/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-kube-api-access-nxqm6\") pod \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") "
Oct 31 04:20:19.548909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548659    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-home\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-home\") pod \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") "
Oct 31 04:20:19.548909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548700    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-dshm\") pod \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") "
Oct 31 04:20:19.548909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548739    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-tmp-dir\") pod \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") "
Oct 31 04:20:19.548909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548747    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-tmp-dir" pod "ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" (UID: "ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.548909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548756    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-home" pod "5fc487b4-195a-42bb-9db6-4aefa3186c6b" (UID: "5fc487b4-195a-42bb-9db6-4aefa3186c6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.548909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548782    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e58cb630-6755-44c8-b068-2bbdb6d4fe15-tls-certs\" (UniqueName: \"kubernetes.io/secret/e58cb630-6755-44c8-b068-2bbdb6d4fe15-tls-certs\") pod \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") "
Oct 31 04:20:19.548909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548905    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-model-cache\") pod \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") "
Oct 31 04:20:19.549362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548947    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-dshm\") pod \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") "
Oct 31 04:20:19.549362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548978    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-home" pod "e58cb630-6755-44c8-b068-2bbdb6d4fe15" (UID: "e58cb630-6755-44c8-b068-2bbdb6d4fe15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.549362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.548991    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-home\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-home\") pod \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") "
Oct 31 04:20:19.549362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.549065    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-model-cache\") pod \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") "
Oct 31 04:20:19.549362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.549099    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-tls-certs\") pod \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\" (UID: \"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63\") "
Oct 31 04:20:19.549362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.549139    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-tmp-dir\") pod \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\" (UID: \"e58cb630-6755-44c8-b068-2bbdb6d4fe15\") "
Oct 31 04:20:19.549362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.549170    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-dshm\") pod \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\" (UID: \"5fc487b4-195a-42bb-9db6-4aefa3186c6b\") "
Oct 31 04:20:19.549362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.549173    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-model-cache" pod "5fc487b4-195a-42bb-9db6-4aefa3186c6b" (UID: "5fc487b4-195a-42bb-9db6-4aefa3186c6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.549362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.549359    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-home" pod "ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" (UID: "ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.549840 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.549775    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.549840 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.549812    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.549840 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.549827    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.550018 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.549840    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.550018 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.549854    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.550139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.550077    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-model-cache" pod "ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" (UID: "ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.551582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.551549    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58cb630-6755-44c8-b068-2bbdb6d4fe15-kube-api-access-s9q5p" pod "e58cb630-6755-44c8-b068-2bbdb6d4fe15" (UID: "e58cb630-6755-44c8-b068-2bbdb6d4fe15"). InnerVolumeSpecName "kube-api-access-s9q5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:20:19.551582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.551567    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e58cb630-6755-44c8-b068-2bbdb6d4fe15-tls-certs" pod "e58cb630-6755-44c8-b068-2bbdb6d4fe15" (UID: "e58cb630-6755-44c8-b068-2bbdb6d4fe15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:20:19.551582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.551553    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fc487b4-195a-42bb-9db6-4aefa3186c6b-tls-certs" pod "5fc487b4-195a-42bb-9db6-4aefa3186c6b" (UID: "5fc487b4-195a-42bb-9db6-4aefa3186c6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:20:19.551809 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.551599    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-kube-api-access-nxqm6" pod "ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" (UID: "ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63"). InnerVolumeSpecName "kube-api-access-nxqm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:20:19.551809 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.551724    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-dshm" pod "ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" (UID: "ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.551809 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.551729    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-tmp-dir" pod "5fc487b4-195a-42bb-9db6-4aefa3186c6b" (UID: "5fc487b4-195a-42bb-9db6-4aefa3186c6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.551926 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.551815    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-model-cache" pod "e58cb630-6755-44c8-b068-2bbdb6d4fe15" (UID: "e58cb630-6755-44c8-b068-2bbdb6d4fe15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.551926 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.551898    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-tmp-dir" pod "e58cb630-6755-44c8-b068-2bbdb6d4fe15" (UID: "e58cb630-6755-44c8-b068-2bbdb6d4fe15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.552272 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.552253    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-dshm" pod "5fc487b4-195a-42bb-9db6-4aefa3186c6b" (UID: "5fc487b4-195a-42bb-9db6-4aefa3186c6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.552351 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.552315    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-dshm" pod "e58cb630-6755-44c8-b068-2bbdb6d4fe15" (UID: "e58cb630-6755-44c8-b068-2bbdb6d4fe15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:20:19.552938 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.552914    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc487b4-195a-42bb-9db6-4aefa3186c6b-kube-api-access-jhx5p" pod "5fc487b4-195a-42bb-9db6-4aefa3186c6b" (UID: "5fc487b4-195a-42bb-9db6-4aefa3186c6b"). InnerVolumeSpecName "kube-api-access-jhx5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:20:19.553386 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.553368    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-tls-certs" pod "ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" (UID: "ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:20:19.579337 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.579314    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:19.651378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.651257    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhx5p\" (UniqueName: \"kubernetes.io/projected/5fc487b4-195a-42bb-9db6-4aefa3186c6b-kube-api-access-jhx5p\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.651378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.651320    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc487b4-195a-42bb-9db6-4aefa3186c6b-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.651378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.651332    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9q5p\" (UniqueName: \"kubernetes.io/projected/e58cb630-6755-44c8-b068-2bbdb6d4fe15-kube-api-access-s9q5p\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.651378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.651342    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxqm6\" (UniqueName: \"kubernetes.io/projected/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-kube-api-access-nxqm6\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.651378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.651353    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.651378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.651362    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.651378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.651370    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e58cb630-6755-44c8-b068-2bbdb6d4fe15-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.651378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.651378    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.651964 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.651387    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.651964 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.651397    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.651964 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.651405    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.651964 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.651414    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e58cb630-6755-44c8-b068-2bbdb6d4fe15-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:19.651964 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:19.651421    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fc487b4-195a-42bb-9db6-4aefa3186c6b-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:20:20.278186 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.278156    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"
Oct 31 04:20:20.278636 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.278193    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5" event={"ID":"ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63","Type":"ContainerDied","Data":"2ea336a312e5f980923d72f63f01bd7c0c943813754ac3ed74955dd3228e20bf"}
Oct 31 04:20:20.278636 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.278249    4900 scope.go:122] "RemoveContainer" containerID="bbbe1d5e2c75d9d438d0cbc73d754b5ffd92b04905668821494dd4ac3de739b3"
Oct 31 04:20:20.279741 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.279724    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"
Oct 31 04:20:20.279886 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.279746    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d" event={"ID":"5fc487b4-195a-42bb-9db6-4aefa3186c6b","Type":"ContainerDied","Data":"b489e377069f87d542573da9ae42c28fe74e1636ff086f6830d899f32037580c"}
Oct 31 04:20:20.282099 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.282073    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp" event={"ID":"e58cb630-6755-44c8-b068-2bbdb6d4fe15","Type":"ContainerDied","Data":"b3d4bce1d00a0567f2d176e5ec5fbbdc4e24796ec80f902d15104ac742688992"}
Oct 31 04:20:20.282224 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.282098    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"
Oct 31 04:20:20.290148 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.290125    4900 scope.go:122] "RemoveContainer" containerID="850fe939d0c5f9b08f02333a7f637faa672b007ed8b8464be971a6b4020532ae"
Oct 31 04:20:20.299720 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.299700    4900 scope.go:122] "RemoveContainer" containerID="8c72a998c6d8a3e085801d5a2677f767073304a31cc8684e55fda64dfba0b9e9"
Oct 31 04:20:20.442689 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.442601    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"]
Oct 31 04:20:20.492518 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.492487    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/distinct-2-w0-7778d0-kserve-55b7c6c859-clml5"]
Oct 31 04:20:20.555872 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.555835    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"]
Oct 31 04:20:20.559035 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.559006    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/simulator-w0-7778d0-kserve-66cf7d94d8-sqxkp"]
Oct 31 04:20:20.668602 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.668569    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"]
Oct 31 04:20:20.673575 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:20.673548    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7778d0/distinct-w0-7778d0-kserve-5559fddd48-vbg7d"]
Oct 31 04:20:21.155129 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:21.155091    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc487b4-195a-42bb-9db6-4aefa3186c6b" path="/var/lib/kubelet/pods/5fc487b4-195a-42bb-9db6-4aefa3186c6b/volumes"
Oct 31 04:20:21.155601 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:21.155585    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e58cb630-6755-44c8-b068-2bbdb6d4fe15" path="/var/lib/kubelet/pods/e58cb630-6755-44c8-b068-2bbdb6d4fe15/volumes"
Oct 31 04:20:21.156046 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:21.156031    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" path="/var/lib/kubelet/pods/ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63/volumes"
Oct 31 04:20:23.180087 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:23.180055    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:24.587066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:24.587029    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:29.431222 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:29.431130    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:32.290801 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:32.290757    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:34.294992 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:34.294954    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:36.388616 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:36.388581    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:36.639842 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:36.639759    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:36.866183 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:36.866143    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:36.984149 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:20:36.984112    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:20:37.380503 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:37.380407    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:39.335367 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:39.335330    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:40.382258 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:40.382226    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:40.608665 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:40.608600    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:41.579767 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:41.579721    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:44.190007 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:44.189975    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:47.292188 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:47.292153    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:47.522340 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:47.522306    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:47.748355 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:47.748326    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:49.074797 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:49.074750    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:54.012632 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:54.012598    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:20:58.891850 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:20:58.891810    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:03.590432 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:03.590396    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:03.803556 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:03.803519    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:06.984308 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:21:06.984264    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:21:07.371014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:07.370934    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:07.798195 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:07.798161    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"]
Oct 31 04:21:07.798389 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:07.798348    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt" podUID="0082baa4-f377-4f35-ac76-961aa2045b86" containerName="maas-api" containerID="cri-o://12eb4b66e0278f5fad79ee1ad49d9e3aba4706145ac489155e63f14ee404c331" gracePeriod=30
Oct 31 04:21:08.003910 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.003802    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:08.072647 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.072618    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 04:21:08.234266 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.234229    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0082baa4-f377-4f35-ac76-961aa2045b86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0082baa4-f377-4f35-ac76-961aa2045b86-openshift-service-ca\") pod \"0082baa4-f377-4f35-ac76-961aa2045b86\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") "
Oct 31 04:21:08.234462 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.234343    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0082baa4-f377-4f35-ac76-961aa2045b86-kube-api-access-vkp55\" (UniqueName: \"kubernetes.io/projected/0082baa4-f377-4f35-ac76-961aa2045b86-kube-api-access-vkp55\") pod \"0082baa4-f377-4f35-ac76-961aa2045b86\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") "
Oct 31 04:21:08.234462 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.234381    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-metrics-certs\") pod \"0082baa4-f377-4f35-ac76-961aa2045b86\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") "
Oct 31 04:21:08.234607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.234558    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-maas-api-tls\") pod \"0082baa4-f377-4f35-ac76-961aa2045b86\" (UID: \"0082baa4-f377-4f35-ac76-961aa2045b86\") "
Oct 31 04:21:08.234672 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.234640    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0082baa4-f377-4f35-ac76-961aa2045b86-openshift-service-ca" pod "0082baa4-f377-4f35-ac76-961aa2045b86" (UID: "0082baa4-f377-4f35-ac76-961aa2045b86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:21:08.235124 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.235098    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0082baa4-f377-4f35-ac76-961aa2045b86-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:08.236746 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.236710    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-maas-api-tls" pod "0082baa4-f377-4f35-ac76-961aa2045b86" (UID: "0082baa4-f377-4f35-ac76-961aa2045b86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:21:08.236874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.236763    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-metrics-certs" pod "0082baa4-f377-4f35-ac76-961aa2045b86" (UID: "0082baa4-f377-4f35-ac76-961aa2045b86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:21:08.236874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.236857    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0082baa4-f377-4f35-ac76-961aa2045b86-kube-api-access-vkp55" pod "0082baa4-f377-4f35-ac76-961aa2045b86" (UID: "0082baa4-f377-4f35-ac76-961aa2045b86"). InnerVolumeSpecName "kube-api-access-vkp55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:21:08.336028 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.335936    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:08.336028 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.335972    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkp55\" (UniqueName: \"kubernetes.io/projected/0082baa4-f377-4f35-ac76-961aa2045b86-kube-api-access-vkp55\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:08.336028 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.335989    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0082baa4-f377-4f35-ac76-961aa2045b86-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:08.556737 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.556688    4900 generic.go:356] "Generic (PLEG): container finished" podID="0082baa4-f377-4f35-ac76-961aa2045b86" containerID="12eb4b66e0278f5fad79ee1ad49d9e3aba4706145ac489155e63f14ee404c331" exitCode=0
Oct 31 04:21:08.556933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.556746    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"
Oct 31 04:21:08.556933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.556764    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt" event={"ID":"0082baa4-f377-4f35-ac76-961aa2045b86","Type":"ContainerDied","Data":"12eb4b66e0278f5fad79ee1ad49d9e3aba4706145ac489155e63f14ee404c331"}
Oct 31 04:21:08.556933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.556836    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt" event={"ID":"0082baa4-f377-4f35-ac76-961aa2045b86","Type":"ContainerDied","Data":"ea73ff0a7a3cda677f353bcc5ff6fa20466aa99920a06cc5c892f52fe5781061"}
Oct 31 04:21:08.556933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.556858    4900 scope.go:122] "RemoveContainer" containerID="12eb4b66e0278f5fad79ee1ad49d9e3aba4706145ac489155e63f14ee404c331"
Oct 31 04:21:08.566754 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.566735    4900 scope.go:122] "RemoveContainer" containerID="12eb4b66e0278f5fad79ee1ad49d9e3aba4706145ac489155e63f14ee404c331"
Oct 31 04:21:08.567156 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:21:08.567135    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12eb4b66e0278f5fad79ee1ad49d9e3aba4706145ac489155e63f14ee404c331\": container with ID starting with 12eb4b66e0278f5fad79ee1ad49d9e3aba4706145ac489155e63f14ee404c331 not found: ID does not exist" containerID="12eb4b66e0278f5fad79ee1ad49d9e3aba4706145ac489155e63f14ee404c331"
Oct 31 04:21:08.567225 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.567166    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12eb4b66e0278f5fad79ee1ad49d9e3aba4706145ac489155e63f14ee404c331"} err="failed to get container status \"12eb4b66e0278f5fad79ee1ad49d9e3aba4706145ac489155e63f14ee404c331\": rpc error: code = NotFound desc = could not find container \"12eb4b66e0278f5fad79ee1ad49d9e3aba4706145ac489155e63f14ee404c331\": container with ID starting with 12eb4b66e0278f5fad79ee1ad49d9e3aba4706145ac489155e63f14ee404c331 not found: ID does not exist"
Oct 31 04:21:08.721302 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.721180    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"]
Oct 31 04:21:08.727646 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:08.727619    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-7778d0-788fd84f94-l8fdt"]
Oct 31 04:21:09.152041 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:09.151949    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0082baa4-f377-4f35-ac76-961aa2045b86" path="/var/lib/kubelet/pods/0082baa4-f377-4f35-ac76-961aa2045b86/volumes"
Oct 31 04:21:09.502112 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:09.502081    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:09.820077 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:09.819992    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:11.226675 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.226646    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"]
Oct 31 04:21:11.227197 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227181    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main"
Oct 31 04:21:11.227248 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227199    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main"
Oct 31 04:21:11.227248 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227211    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c0e37b79-c673-4bb9-a740-eda25789449a" containerName="istio-proxy"
Oct 31 04:21:11.227248 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227216    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e37b79-c673-4bb9-a740-eda25789449a" containerName="istio-proxy"
Oct 31 04:21:11.227248 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227229    4900 cpu_manager.go:414] "containerMap: removing container" podUID="5fc487b4-195a-42bb-9db6-4aefa3186c6b" containerName="main"
Oct 31 04:21:11.227248 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227235    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc487b4-195a-42bb-9db6-4aefa3186c6b" containerName="main"
Oct 31 04:21:11.227248 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227246    4900 cpu_manager.go:414] "containerMap: removing container" podUID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerName="istio-proxy"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227254    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerName="istio-proxy"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227264    4900 cpu_manager.go:414] "containerMap: removing container" podUID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerName="istio-proxy"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227269    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerName="istio-proxy"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227279    4900 cpu_manager.go:414] "containerMap: removing container" podUID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerName="istio-proxy"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227284    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerName="istio-proxy"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227293    4900 cpu_manager.go:414] "containerMap: removing container" podUID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerName="istio-proxy"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227297    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerName="istio-proxy"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227306    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0082baa4-f377-4f35-ac76-961aa2045b86" containerName="maas-api"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227311    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0082baa4-f377-4f35-ac76-961aa2045b86" containerName="maas-api"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227320    4900 cpu_manager.go:414] "containerMap: removing container" podUID="e58cb630-6755-44c8-b068-2bbdb6d4fe15" containerName="main"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227325    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58cb630-6755-44c8-b068-2bbdb6d4fe15" containerName="main"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227345    4900 cpu_manager.go:414] "containerMap: removing container" podUID="2e5fe713-f05f-4052-8621-ac82812d0000" containerName="maas-api"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227350    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e5fe713-f05f-4052-8621-ac82812d0000" containerName="maas-api"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227365    4900 cpu_manager.go:414] "containerMap: removing container" podUID="3e59d7a0-72ed-4065-9538-cc15b531b5fe" containerName="maas-api"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227369    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e59d7a0-72ed-4065-9538-cc15b531b5fe" containerName="maas-api"
Oct 31 04:21:11.227444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227445    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e5fe713-f05f-4052-8621-ac82812d0000" containerName="maas-api"
Oct 31 04:21:11.227968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227456    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e37b79-c673-4bb9-a740-eda25789449a" containerName="istio-proxy"
Oct 31 04:21:11.227968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227462    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e59d7a0-72ed-4065-9538-cc15b531b5fe" containerName="maas-api"
Oct 31 04:21:11.227968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227468    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec7f8536-2b67-47fe-8f8e-7dd6ea39bb63" containerName="main"
Oct 31 04:21:11.227968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227474    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e37b79-c673-4bb9-a740-eda25789449a" containerName="istio-proxy"
Oct 31 04:21:11.227968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227479    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc487b4-195a-42bb-9db6-4aefa3186c6b" containerName="main"
Oct 31 04:21:11.227968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227484    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerName="istio-proxy"
Oct 31 04:21:11.227968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227492    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerName="istio-proxy"
Oct 31 04:21:11.227968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227501    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed827b7-7ac6-412d-ade0-70ce0d8cc399" containerName="istio-proxy"
Oct 31 04:21:11.227968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227507    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b94bdc-4427-400c-ad3f-5032aa82a83f" containerName="istio-proxy"
Oct 31 04:21:11.227968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227513    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0082baa4-f377-4f35-ac76-961aa2045b86" containerName="maas-api"
Oct 31 04:21:11.227968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.227520    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58cb630-6755-44c8-b068-2bbdb6d4fe15" containerName="main"
Oct 31 04:21:11.232045 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.232028    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.285995 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.285969    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7778d0-openshift-default-dockercfg-c48rb\""
Oct 31 04:21:11.338515 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.338481    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"]
Oct 31 04:21:11.369576 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.369538    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.369769 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.369591    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.369769 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.369726    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.369925 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.369779    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.369925 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.369814    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.369925 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.369911    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.370067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.369969    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.370067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.370007    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.370067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.370045    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v27bp\" (UniqueName: \"kubernetes.io/projected/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-kube-api-access-v27bp\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.471677 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.471636    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.471925 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.471688    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.471925 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.471716    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v27bp\" (UniqueName: \"kubernetes.io/projected/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-kube-api-access-v27bp\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.471925 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.471760    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.471925 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.471838    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.471925 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.471895    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.472211 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.471944    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.472211 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.471979    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.472211 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.472061    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.472211 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.472130    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-credential-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.472442 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.472262    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-workload-socket\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.472442 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.472339    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-workload-certs\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.472550 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.472468    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istiod-ca-cert\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.472550 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.472491    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-data\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.474321 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.474298    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-envoy\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.474433 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.474414    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-podinfo\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.528453 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.528414    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-token\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.528611 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.528471    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v27bp\" (UniqueName: \"kubernetes.io/projected/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-kube-api-access-v27bp\") pod \"e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") " pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.545330 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.545303    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:11.959744 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:21:11.959703    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9f3a8d_1f31_4044_840f_bfe6df1afbc2.slice/crio-9c18ada39584da49e57a7b9b58577bfba611726622cc653addb334a125048625 WatchSource:0}: Error finding container 9c18ada39584da49e57a7b9b58577bfba611726622cc653addb334a125048625: Status 404 returned error can't find the container with id 9c18ada39584da49e57a7b9b58577bfba611726622cc653addb334a125048625
Oct 31 04:21:11.961070 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.961049    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"]
Oct 31 04:21:11.962303 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.962268    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:21:11.962422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.962342    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:21:11.962422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:11.962388    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:21:12.580765 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:12.580729    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx" event={"ID":"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2","Type":"ContainerStarted","Data":"0e64783cf123352eeb3fdb5788cfc8f3c9289c613c45946f488f6ace60a146ec"}
Oct 31 04:21:12.581197 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:12.580809    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx" event={"ID":"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2","Type":"ContainerStarted","Data":"9c18ada39584da49e57a7b9b58577bfba611726622cc653addb334a125048625"}
Oct 31 04:21:12.699502 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:12.699452    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx" podStartSLOduration=1.699439637 podStartE2EDuration="1.699439637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:21:12.698559512 +0000 UTC m=+4425.664892855" watchObservedRunningTime="2026-10-31 04:21:12.699439637 +0000 UTC m=+4425.665772981"
Oct 31 04:21:13.553900 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:13.553868    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:13.554258 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:13.554236    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 31 04:21:13.554427 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:13.554402    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx" podUID="ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 31 04:21:14.557811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:14.557069    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 31 04:21:14.557811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:14.557141    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx" podUID="ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 31 04:21:14.621913 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:14.621872    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:14.860199 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:14.860117    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:15.552281 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:15.552238    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 31 04:21:15.552578 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:15.552546    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx" podUID="ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 31 04:21:16.327833 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:16.327771    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:16.548463 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:16.546535    4900 patch_prober.go:29] interesting pod/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body=
Oct 31 04:21:16.548463 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:16.546601    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx" podUID="ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused"
Oct 31 04:21:16.764654 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:16.764611    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"]
Oct 31 04:21:16.764959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:16.764930    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx" podUID="ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" containerName="istio-proxy" containerID="cri-o://0e64783cf123352eeb3fdb5788cfc8f3c9289c613c45946f488f6ace60a146ec" gracePeriod=30
Oct 31 04:21:16.859593 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:16.859544    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:17.213614 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:17.213515    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:19.488570 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:19.488536    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:19.714568 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:19.714530    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:20.012869 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:20.012835    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:23.241425 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.241391    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:23.646371 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.646152    4900 generic.go:356] "Generic (PLEG): container finished" podID="ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" containerID="0e64783cf123352eeb3fdb5788cfc8f3c9289c613c45946f488f6ace60a146ec" exitCode=0
Oct 31 04:21:23.646371 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.646304    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx" event={"ID":"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2","Type":"ContainerDied","Data":"0e64783cf123352eeb3fdb5788cfc8f3c9289c613c45946f488f6ace60a146ec"}
Oct 31 04:21:23.727412 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.727384    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:23.842209 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.842119    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-workload-socket\") pod \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") "
Oct 31 04:21:23.842209 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.842167    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-kube-api-access-v27bp\" (UniqueName: \"kubernetes.io/projected/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-kube-api-access-v27bp\") pod \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") "
Oct 31 04:21:23.842209 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.842193    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-podinfo\") pod \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") "
Oct 31 04:21:23.842514 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.842216    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-workload-certs\") pod \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") "
Oct 31 04:21:23.842514 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.842243    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-envoy\") pod \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") "
Oct 31 04:21:23.842514 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.842343    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-credential-socket\") pod \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") "
Oct 31 04:21:23.842514 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.842409    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-workload-socket" pod "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" (UID: "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:21:23.842514 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.842432    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istiod-ca-cert\") pod \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") "
Oct 31 04:21:23.842514 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.842481    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-data\") pod \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") "
Oct 31 04:21:23.842899 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.842587    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-workload-certs" pod "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" (UID: "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:21:23.842899 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.842619    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-credential-socket" pod "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" (UID: "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:21:23.842899 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.842695    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-token\" (UniqueName: \"kubernetes.io/projected/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-token\") pod \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\" (UID: \"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2\") "
Oct 31 04:21:23.842899 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.842885    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-data" pod "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" (UID: "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:21:23.843108 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.842895    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istiod-ca-cert" pod "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" (UID: "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:21:23.843240 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.843221    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:23.843302 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.843246    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:23.843302 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.843260    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:23.843302 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.843274    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:23.843302 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.843289    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:23.844439 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.844410    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-envoy" pod "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" (UID: "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:21:23.844619 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.844601    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-token" pod "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" (UID: "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:21:23.844862 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.844843    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-kube-api-access-v27bp" pod "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" (UID: "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2"). InnerVolumeSpecName "kube-api-access-v27bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:21:23.844862 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.844849    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-podinfo" pod "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" (UID: "ac9f3a8d-1f31-4044-840f-bfe6df1afbc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:21:23.944398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.944356    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v27bp\" (UniqueName: \"kubernetes.io/projected/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-kube-api-access-v27bp\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:23.944398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.944391    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:23.944398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.944404    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:23.944660 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:23.944412    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:21:24.653644 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:24.653606    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx" event={"ID":"ac9f3a8d-1f31-4044-840f-bfe6df1afbc2","Type":"ContainerDied","Data":"9c18ada39584da49e57a7b9b58577bfba611726622cc653addb334a125048625"}
Oct 31 04:21:24.653644 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:24.653647    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"
Oct 31 04:21:24.654193 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:24.653678    4900 scope.go:122] "RemoveContainer" containerID="0e64783cf123352eeb3fdb5788cfc8f3c9289c613c45946f488f6ace60a146ec"
Oct 31 04:21:24.869645 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:24.869613    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"]
Oct 31 04:21:24.876966 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:24.876937    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:24.918894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:24.918817    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7778d0-openshift-default-56c8f7f987-tthrx"]
Oct 31 04:21:25.152987 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:25.152953    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" path="/var/lib/kubelet/pods/ac9f3a8d-1f31-4044-840f-bfe6df1afbc2/volumes"
Oct 31 04:21:29.785986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:29.785950    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:34.662389 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:34.662349    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:36.984439 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:21:36.984402    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:21:39.582632 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:39.582577    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:44.569669 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:44.569633    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:49.411913 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:49.411869    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:54.283178 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:54.283138    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:21:59.194112 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:21:59.194077    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:00.208035 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:00.207997    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:00.425919 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:00.425885    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:04.081375 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:04.081344    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:06.984685 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:22:06.984643    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:22:07.707707 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:07.707671    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:10.051205 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:10.051173    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:13.871031 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:13.870995    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:18.641558 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:18.641524    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:23.339389 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:23.339357    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:28.102282 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:28.102244    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:28.943877 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:28.943848    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:22:28.951534 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:28.951510    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:22:32.877320 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:32.877290    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:34.092768 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.092734    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:34.606175 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.606137    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"]
Oct 31 04:22:34.606776 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.606755    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" containerName="istio-proxy"
Oct 31 04:22:34.606854 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.606778    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" containerName="istio-proxy"
Oct 31 04:22:34.606854 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.606815    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c0e37b79-c673-4bb9-a740-eda25789449a" containerName="istio-proxy"
Oct 31 04:22:34.606854 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.606821    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e37b79-c673-4bb9-a740-eda25789449a" containerName="istio-proxy"
Oct 31 04:22:34.606958 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.606904    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac9f3a8d-1f31-4044-840f-bfe6df1afbc2" containerName="istio-proxy"
Oct 31 04:22:34.610207 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.610191    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"
Oct 31 04:22:34.662736 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.662696    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"]
Oct 31 04:22:34.709912 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.709880    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:22:34.710096 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.709924    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:22:34.712441 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.712421    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3113317-83f1-480e-8522-dba71d8bb657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40561656-clhld\" (UID: \"b3113317-83f1-480e-8522-dba71d8bb657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"
Oct 31 04:22:34.712518 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.712475    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqqxm\" (UniqueName: \"kubernetes.io/projected/b3113317-83f1-480e-8522-dba71d8bb657-kube-api-access-fqqxm\") pod \"maas-api-revoke-sub-40561656-clhld\" (UID: \"b3113317-83f1-480e-8522-dba71d8bb657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"
Oct 31 04:22:34.712597 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.712566    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3113317-83f1-480e-8522-dba71d8bb657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40561656-clhld\" (UID: \"b3113317-83f1-480e-8522-dba71d8bb657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"
Oct 31 04:22:34.814077 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.814040    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3113317-83f1-480e-8522-dba71d8bb657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40561656-clhld\" (UID: \"b3113317-83f1-480e-8522-dba71d8bb657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"
Oct 31 04:22:34.814301 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.814134    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqqxm\" (UniqueName: \"kubernetes.io/projected/b3113317-83f1-480e-8522-dba71d8bb657-kube-api-access-fqqxm\") pod \"maas-api-revoke-sub-40561656-clhld\" (UID: \"b3113317-83f1-480e-8522-dba71d8bb657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"
Oct 31 04:22:34.814301 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.814160    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3113317-83f1-480e-8522-dba71d8bb657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40561656-clhld\" (UID: \"b3113317-83f1-480e-8522-dba71d8bb657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"
Oct 31 04:22:34.814884 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.814862    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3113317-83f1-480e-8522-dba71d8bb657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40561656-clhld\" (UID: \"b3113317-83f1-480e-8522-dba71d8bb657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"
Oct 31 04:22:34.816514 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.816493    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3113317-83f1-480e-8522-dba71d8bb657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40561656-clhld\" (UID: \"b3113317-83f1-480e-8522-dba71d8bb657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"
Oct 31 04:22:34.870940 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.870864    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqqxm\" (UniqueName: \"kubernetes.io/projected/b3113317-83f1-480e-8522-dba71d8bb657-kube-api-access-fqqxm\") pod \"maas-api-revoke-sub-40561656-clhld\" (UID: \"b3113317-83f1-480e-8522-dba71d8bb657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"
Oct 31 04:22:34.922815 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:34.922767    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"
Oct 31 04:22:35.289396 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:22:35.289361    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3113317_83f1_480e_8522_dba71d8bb657.slice/crio-1d41e51a1972b23c246c2eb61212ba8d29266f44e83fdc441ecf3a182b414a68 WatchSource:0}: Error finding container 1d41e51a1972b23c246c2eb61212ba8d29266f44e83fdc441ecf3a182b414a68: Status 404 returned error can't find the container with id 1d41e51a1972b23c246c2eb61212ba8d29266f44e83fdc441ecf3a182b414a68
Oct 31 04:22:35.292067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:35.292047    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"]
Oct 31 04:22:36.087625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:36.087588    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld" event={"ID":"b3113317-83f1-480e-8522-dba71d8bb657","Type":"ContainerStarted","Data":"63a4ea044c1577f38249a267b2727dac847f181218460521cc2b18486c1ebb3d"}
Oct 31 04:22:36.087851 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:36.087666    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld" event={"ID":"b3113317-83f1-480e-8522-dba71d8bb657","Type":"ContainerStarted","Data":"1d41e51a1972b23c246c2eb61212ba8d29266f44e83fdc441ecf3a182b414a68"}
Oct 31 04:22:36.202541 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:36.202487    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld" podStartSLOduration=2.202472577 podStartE2EDuration="2.202472577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:22:36.200633354 +0000 UTC m=+4509.166966698" watchObservedRunningTime="2026-10-31 04:22:36.202472577 +0000 UTC m=+4509.168805899"
Oct 31 04:22:36.983758 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:22:36.983720    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:22:37.094529 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:37.094497    4900 generic.go:356] "Generic (PLEG): container finished" podID="b3113317-83f1-480e-8522-dba71d8bb657" containerID="63a4ea044c1577f38249a267b2727dac847f181218460521cc2b18486c1ebb3d" exitCode=0
Oct 31 04:22:37.094705 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:37.094590    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld" event={"ID":"b3113317-83f1-480e-8522-dba71d8bb657","Type":"ContainerDied","Data":"63a4ea044c1577f38249a267b2727dac847f181218460521cc2b18486c1ebb3d"}
Oct 31 04:22:37.670336 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:37.670290    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:38.241468 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:38.241438    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"
Oct 31 04:22:38.256634 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:38.256605    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3113317-83f1-480e-8522-dba71d8bb657-kube-api-access-fqqxm\" (UniqueName: \"kubernetes.io/projected/b3113317-83f1-480e-8522-dba71d8bb657-kube-api-access-fqqxm\") pod \"b3113317-83f1-480e-8522-dba71d8bb657\" (UID: \"b3113317-83f1-480e-8522-dba71d8bb657\") "
Oct 31 04:22:38.256820 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:38.256753    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3113317-83f1-480e-8522-dba71d8bb657-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3113317-83f1-480e-8522-dba71d8bb657-maas-api-openshift-ca\") pod \"b3113317-83f1-480e-8522-dba71d8bb657\" (UID: \"b3113317-83f1-480e-8522-dba71d8bb657\") "
Oct 31 04:22:38.256870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:38.256829    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3113317-83f1-480e-8522-dba71d8bb657-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3113317-83f1-480e-8522-dba71d8bb657-maas-api-xks-ca\") pod \"b3113317-83f1-480e-8522-dba71d8bb657\" (UID: \"b3113317-83f1-480e-8522-dba71d8bb657\") "
Oct 31 04:22:38.257115 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:38.257092    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3113317-83f1-480e-8522-dba71d8bb657-maas-api-openshift-ca" pod "b3113317-83f1-480e-8522-dba71d8bb657" (UID: "b3113317-83f1-480e-8522-dba71d8bb657"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:22:38.257322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:38.257301    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3113317-83f1-480e-8522-dba71d8bb657-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:38.258867 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:38.258817    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3113317-83f1-480e-8522-dba71d8bb657-maas-api-xks-ca" pod "b3113317-83f1-480e-8522-dba71d8bb657" (UID: "b3113317-83f1-480e-8522-dba71d8bb657"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:22:38.258972 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:38.258897    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3113317-83f1-480e-8522-dba71d8bb657-kube-api-access-fqqxm" pod "b3113317-83f1-480e-8522-dba71d8bb657" (UID: "b3113317-83f1-480e-8522-dba71d8bb657"). InnerVolumeSpecName "kube-api-access-fqqxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:22:38.358226 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:38.358129    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3113317-83f1-480e-8522-dba71d8bb657-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:38.358226 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:38.358164    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqqxm\" (UniqueName: \"kubernetes.io/projected/b3113317-83f1-480e-8522-dba71d8bb657-kube-api-access-fqqxm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:39.107634 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:39.107557    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"
Oct 31 04:22:39.107853 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:39.107551    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld" event={"ID":"b3113317-83f1-480e-8522-dba71d8bb657","Type":"ContainerDied","Data":"1d41e51a1972b23c246c2eb61212ba8d29266f44e83fdc441ecf3a182b414a68"}
Oct 31 04:22:39.107853 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:39.107720    4900 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d41e51a1972b23c246c2eb61212ba8d29266f44e83fdc441ecf3a182b414a68"
Oct 31 04:22:39.198290 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:39.198253    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:40.982437 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:40.982405    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:42.054228 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:42.054200    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:42.469393 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:42.469297    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:46.342976 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.342943    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"]
Oct 31 04:22:46.343459 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.343446    4900 cpu_manager.go:414] "containerMap: removing container" podUID="b3113317-83f1-480e-8522-dba71d8bb657" containerName="revoke-keys"
Oct 31 04:22:46.343504 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.343459    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3113317-83f1-480e-8522-dba71d8bb657" containerName="revoke-keys"
Oct 31 04:22:46.343539 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.343525    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3113317-83f1-480e-8522-dba71d8bb657" containerName="revoke-keys"
Oct 31 04:22:46.346810 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.346764    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"
Oct 31 04:22:46.450756 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.450725    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:22:46.450946 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.450859    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:22:46.451227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.451208    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dc10fc5-nx5dd\" (UID: \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"
Oct 31 04:22:46.451278 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.451238    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thzvr\" (UniqueName: \"kubernetes.io/projected/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-kube-api-access-thzvr\") pod \"maas-api-revoke-sub-8dc10fc5-nx5dd\" (UID: \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"
Oct 31 04:22:46.451422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.451397    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dc10fc5-nx5dd\" (UID: \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"
Oct 31 04:22:46.453366 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.453349    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"]
Oct 31 04:22:46.552764 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.552725    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dc10fc5-nx5dd\" (UID: \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"
Oct 31 04:22:46.552980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.552929    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dc10fc5-nx5dd\" (UID: \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"
Oct 31 04:22:46.552980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.552966    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thzvr\" (UniqueName: \"kubernetes.io/projected/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-kube-api-access-thzvr\") pod \"maas-api-revoke-sub-8dc10fc5-nx5dd\" (UID: \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"
Oct 31 04:22:46.553428 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.553406    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dc10fc5-nx5dd\" (UID: \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"
Oct 31 04:22:46.555148 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.555129    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dc10fc5-nx5dd\" (UID: \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"
Oct 31 04:22:46.610454 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.610385    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thzvr\" (UniqueName: \"kubernetes.io/projected/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-kube-api-access-thzvr\") pod \"maas-api-revoke-sub-8dc10fc5-nx5dd\" (UID: \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"
Oct 31 04:22:46.659621 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.659588    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"
Oct 31 04:22:46.902445 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:46.902314    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:47.025766 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:22:47.025732    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4e29f3_e344_40ea_b9e7_497bef1b2c24.slice/crio-7a2514626bafafbe1b676d4823c317281ecf0330fea5c54f0a13844af8b1d085 WatchSource:0}: Error finding container 7a2514626bafafbe1b676d4823c317281ecf0330fea5c54f0a13844af8b1d085: Status 404 returned error can't find the container with id 7a2514626bafafbe1b676d4823c317281ecf0330fea5c54f0a13844af8b1d085
Oct 31 04:22:47.028693 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:47.028658    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"]
Oct 31 04:22:47.154125 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:47.154042    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd" event={"ID":"ca4e29f3-e344-40ea-b9e7-497bef1b2c24","Type":"ContainerStarted","Data":"7a2514626bafafbe1b676d4823c317281ecf0330fea5c54f0a13844af8b1d085"}
Oct 31 04:22:47.290612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:47.290580    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:47.814570 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:47.814537    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"]
Oct 31 04:22:47.857933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:47.857898    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:48.160096 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.160008    4900 generic.go:356] "Generic (PLEG): container finished" podID="ca4e29f3-e344-40ea-b9e7-497bef1b2c24" containerID="e324d98e48fdda0bd2f0cb6d93148656895421d052cb1c277363c14b10e54afe" exitCode=0
Oct 31 04:22:48.160096 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.160079    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd" event={"ID":"ca4e29f3-e344-40ea-b9e7-497bef1b2c24","Type":"ContainerDied","Data":"e324d98e48fdda0bd2f0cb6d93148656895421d052cb1c277363c14b10e54afe"}
Oct 31 04:22:48.183925 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.183881    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:48.318940 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.318904    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"]
Oct 31 04:22:48.322764 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.322745    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.405649 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.405612    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-workload-certs\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.405884 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.405729    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e17b74-45d2-4d36-842f-1c1868c501e0-istiod-ca-cert\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.405884 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.405767    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-data\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.405884 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.405822    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-workload-socket\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.405884 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.405882    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-podinfo\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.406139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.405906    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-envoy\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.406139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.405948    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-679h8\" (UniqueName: \"kubernetes.io/projected/36e17b74-45d2-4d36-842f-1c1868c501e0-kube-api-access-679h8\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.406139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.405982    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-token\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.406139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.406036    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-credential-socket\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.406658 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.406633    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:48.406780 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.406755    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"]
Oct 31 04:22:48.455512 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.455430    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-26c74d32-openshift-default-dockercfg-6t24z\""
Oct 31 04:22:48.510184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.507646    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-workload-certs\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.510184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.507761    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e17b74-45d2-4d36-842f-1c1868c501e0-istiod-ca-cert\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.510184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.507840    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-data\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.510184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.507885    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-workload-socket\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.510184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.507949    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-podinfo\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.510184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.507975    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-envoy\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.510184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.508020    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-679h8\" (UniqueName: \"kubernetes.io/projected/36e17b74-45d2-4d36-842f-1c1868c501e0-kube-api-access-679h8\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.510184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.508057    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-token\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.510184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.508115    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-credential-socket\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.510184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.508689    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-credential-socket\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.510184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.508992    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-workload-certs\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.510184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.509626    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e17b74-45d2-4d36-842f-1c1868c501e0-istiod-ca-cert\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.510184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.510100    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-data\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.514014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.510952    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-workload-socket\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.514014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.513461    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-envoy\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.515317 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.515287    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-podinfo\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.567734 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.567698    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-token\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.568026 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.568002    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-679h8\" (UniqueName: \"kubernetes.io/projected/36e17b74-45d2-4d36-842f-1c1868c501e0-kube-api-access-679h8\") pod \"e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") " pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.636342 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.636307    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:48.714972 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.714900    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:48.917573 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:48.917544    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:49.055121 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:22:49.055077    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e17b74_45d2_4d36_842f_1c1868c501e0.slice/crio-ba2b6aae5069b7515e4616d8825c4fe3042ddb984e349763ee55e14645dc71bc WatchSource:0}: Error finding container ba2b6aae5069b7515e4616d8825c4fe3042ddb984e349763ee55e14645dc71bc: Status 404 returned error can't find the container with id ba2b6aae5069b7515e4616d8825c4fe3042ddb984e349763ee55e14645dc71bc
Oct 31 04:22:49.057961 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.057927    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:22:49.058043 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.057994    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:22:49.058043 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.058022    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:22:49.058348 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.058316    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"]
Oct 31 04:22:49.186808 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.186529    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj" event={"ID":"36e17b74-45d2-4d36-842f-1c1868c501e0","Type":"ContainerStarted","Data":"5f2f03666608e01fe6c38cf054d95fa8019df4b1e303defca2b7bf9ab7911e83"}
Oct 31 04:22:49.186808 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.186593    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj" event={"ID":"36e17b74-45d2-4d36-842f-1c1868c501e0","Type":"ContainerStarted","Data":"ba2b6aae5069b7515e4616d8825c4fe3042ddb984e349763ee55e14645dc71bc"}
Oct 31 04:22:49.303997 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.303891    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj" podStartSLOduration=1.303873895 podStartE2EDuration="1.303873895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:22:49.301302165 +0000 UTC m=+4522.267635509" watchObservedRunningTime="2026-10-31 04:22:49.303873895 +0000 UTC m=+4522.270207241"
Oct 31 04:22:49.329598 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.329567    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"
Oct 31 04:22:49.418903 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.418870    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-maas-api-openshift-ca\") pod \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\" (UID: \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\") "
Oct 31 04:22:49.419103 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.418949    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-maas-api-xks-ca\") pod \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\" (UID: \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\") "
Oct 31 04:22:49.419176 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.419150    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-kube-api-access-thzvr\" (UniqueName: \"kubernetes.io/projected/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-kube-api-access-thzvr\") pod \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\" (UID: \"ca4e29f3-e344-40ea-b9e7-497bef1b2c24\") "
Oct 31 04:22:49.419235 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.419211    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-maas-api-openshift-ca" pod "ca4e29f3-e344-40ea-b9e7-497bef1b2c24" (UID: "ca4e29f3-e344-40ea-b9e7-497bef1b2c24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:22:49.419556 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.419539    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:49.420934 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.420913    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-maas-api-xks-ca" pod "ca4e29f3-e344-40ea-b9e7-497bef1b2c24" (UID: "ca4e29f3-e344-40ea-b9e7-497bef1b2c24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:22:49.421114 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.421090    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-kube-api-access-thzvr" pod "ca4e29f3-e344-40ea-b9e7-497bef1b2c24" (UID: "ca4e29f3-e344-40ea-b9e7-497bef1b2c24"). InnerVolumeSpecName "kube-api-access-thzvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:22:49.521333 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.521276    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thzvr\" (UniqueName: \"kubernetes.io/projected/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-kube-api-access-thzvr\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:49.521333 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.521322    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca4e29f3-e344-40ea-b9e7-497bef1b2c24-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:22:49.636811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.636762    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:49.692153 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.692124    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"]
Oct 31 04:22:49.692649 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.692636    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ca4e29f3-e344-40ea-b9e7-497bef1b2c24" containerName="revoke-keys"
Oct 31 04:22:49.692706 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.692651    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca4e29f3-e344-40ea-b9e7-497bef1b2c24" containerName="revoke-keys"
Oct 31 04:22:49.692756 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.692746    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca4e29f3-e344-40ea-b9e7-497bef1b2c24" containerName="revoke-keys"
Oct 31 04:22:49.696054 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.696038    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.748782 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.748730    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2339ed\"/\"kube-root-ca.crt\""
Oct 31 04:22:49.748981 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.748778    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2339ed\"/\"openshift-service-ca.crt\""
Oct 31 04:22:49.748981 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.748767    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2339ed\"/\"rate-test-model-2339ed-kserve-self-signed-certs\""
Oct 31 04:22:49.748981 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.748828    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2339ed\"/\"default-dockercfg-cxp4c\""
Oct 31 04:22:49.749415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.749393    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"]
Oct 31 04:22:49.766869 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.766643    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:49.824585 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.824554    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-tls-certs\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.824780 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.824609    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-model-cache\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.824780 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.824744    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-home\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.824909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.824842    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lmck\" (UniqueName: \"kubernetes.io/projected/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-kube-api-access-5lmck\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.824909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.824879    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-dshm\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.824977 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.824943    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-tmp-dir\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.926732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.926646    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lmck\" (UniqueName: \"kubernetes.io/projected/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-kube-api-access-5lmck\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.926732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.926696    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-dshm\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.926732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.926718    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-tmp-dir\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.927357 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.926813    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-tls-certs\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.927357 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.926849    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-model-cache\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.927357 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.926907    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-home\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.927357 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.927219    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-tmp-dir\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.927357 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.927295    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-model-cache\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.927357 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.927317    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-home\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.929163 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.929135    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-dshm\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.929456 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.929435    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-tls-certs\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:49.984629 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:49.984591    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lmck\" (UniqueName: \"kubernetes.io/projected/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-kube-api-access-5lmck\") pod \"rate-test-model-2339ed-kserve-9974f87b5-7vc98\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") " pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:50.008385 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:50.008347    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:50.189933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:50.189843    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"
Oct 31 04:22:50.189933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:50.189887    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd" event={"ID":"ca4e29f3-e344-40ea-b9e7-497bef1b2c24","Type":"ContainerDied","Data":"7a2514626bafafbe1b676d4823c317281ecf0330fea5c54f0a13844af8b1d085"}
Oct 31 04:22:50.189933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:50.189925    4900 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a2514626bafafbe1b676d4823c317281ecf0330fea5c54f0a13844af8b1d085"
Oct 31 04:22:50.376072 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:22:50.376039    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podade0cb97_d48a_4a37_9a3e_96ea531fbf46.slice/crio-036ff4ff382a4adef6c46022ff38128093db2d763af8447fe4a2debe1a47c7a2 WatchSource:0}: Error finding container 036ff4ff382a4adef6c46022ff38128093db2d763af8447fe4a2debe1a47c7a2: Status 404 returned error can't find the container with id 036ff4ff382a4adef6c46022ff38128093db2d763af8447fe4a2debe1a47c7a2
Oct 31 04:22:50.378095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:50.378068    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"]
Oct 31 04:22:50.642271 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:50.642238    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:51.205868 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:51.205834    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98" event={"ID":"ade0cb97-d48a-4a37-9a3e-96ea531fbf46","Type":"ContainerStarted","Data":"90428b3af54d6d3fee71e3933f400386e2fa92d563c9873a34d24a784a427d03"}
Oct 31 04:22:51.206394 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:51.205875    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98" event={"ID":"ade0cb97-d48a-4a37-9a3e-96ea531fbf46","Type":"ContainerStarted","Data":"036ff4ff382a4adef6c46022ff38128093db2d763af8447fe4a2debe1a47c7a2"}
Oct 31 04:22:51.206394 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:51.206134    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:51.207160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:51.207138    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:22:51.319209 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:51.319152    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98" podStartSLOduration=2.319136256 podStartE2EDuration="2.319136256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:22:51.318082062 +0000 UTC m=+4524.284415411" watchObservedRunningTime="2026-10-31 04:22:51.319136256 +0000 UTC m=+4524.285469600"
Oct 31 04:22:52.009556 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:52.009528    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:22:52.411928 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:52.411850    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:52.627637 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:52.627604    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:52.815038 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:52.814961    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9" podUID="25920611-22a3-431c-b9cd-140eddaee0d0" containerName="payload-pre-processing" containerID="cri-o://8f5d6dd003f40db5fe7b43bb53682ed56b125e6820269f7ca824557cc56f73f9" gracePeriod=25
Oct 31 04:22:52.959128 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:52.959098    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:55.712498 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:55.712450    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:55.927093 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:55.927061    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:55.995865 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:55.995827    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"]
Oct 31 04:22:55.999869 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:55.999848    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"
Oct 31 04:22:56.003590 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.003568    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtfv2\" (UniqueName: \"kubernetes.io/projected/8722d22a-ef10-4b87-9616-80299e5a6279-kube-api-access-jtfv2\") pod \"payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr\" (UID: \"8722d22a-ef10-4b87-9616-80299e5a6279\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"
Oct 31 04:22:56.003807 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.003778    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8722d22a-ef10-4b87-9616-80299e5a6279-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr\" (UID: \"8722d22a-ef10-4b87-9616-80299e5a6279\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"
Oct 31 04:22:56.052158 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.052129    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"]
Oct 31 04:22:56.099545 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.099420    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-26c74d32\""
Oct 31 04:22:56.099545 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.099498    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-26c74d32-dockercfg-s9l9s\""
Oct 31 04:22:56.105145 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.104751    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8722d22a-ef10-4b87-9616-80299e5a6279-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr\" (UID: \"8722d22a-ef10-4b87-9616-80299e5a6279\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"
Oct 31 04:22:56.105145 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.104870    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtfv2\" (UniqueName: \"kubernetes.io/projected/8722d22a-ef10-4b87-9616-80299e5a6279-kube-api-access-jtfv2\") pod \"payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr\" (UID: \"8722d22a-ef10-4b87-9616-80299e5a6279\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"
Oct 31 04:22:56.108381 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.108351    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8722d22a-ef10-4b87-9616-80299e5a6279-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr\" (UID: \"8722d22a-ef10-4b87-9616-80299e5a6279\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"
Oct 31 04:22:56.144275 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.144247    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"]
Oct 31 04:22:56.148569 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.148547    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"
Oct 31 04:22:56.161860 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.161832    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtfv2\" (UniqueName: \"kubernetes.io/projected/8722d22a-ef10-4b87-9616-80299e5a6279-kube-api-access-jtfv2\") pod \"payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr\" (UID: \"8722d22a-ef10-4b87-9616-80299e5a6279\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"
Oct 31 04:22:56.199725 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.199696    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"]
Oct 31 04:22:56.205976 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.205949    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cd9r\" (UniqueName: \"kubernetes.io/projected/0e83740c-ab5a-49f8-9d97-1fadd09b65e4-kube-api-access-5cd9r\") pod \"payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz\" (UID: \"0e83740c-ab5a-49f8-9d97-1fadd09b65e4\") " pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"
Oct 31 04:22:56.206155 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.206135    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e83740c-ab5a-49f8-9d97-1fadd09b65e4-plugins-config-volume\") pod \"payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz\" (UID: \"0e83740c-ab5a-49f8-9d97-1fadd09b65e4\") " pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"
Oct 31 04:22:56.307470 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.307376    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cd9r\" (UniqueName: \"kubernetes.io/projected/0e83740c-ab5a-49f8-9d97-1fadd09b65e4-kube-api-access-5cd9r\") pod \"payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz\" (UID: \"0e83740c-ab5a-49f8-9d97-1fadd09b65e4\") " pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"
Oct 31 04:22:56.307663 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.307534    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e83740c-ab5a-49f8-9d97-1fadd09b65e4-plugins-config-volume\") pod \"payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz\" (UID: \"0e83740c-ab5a-49f8-9d97-1fadd09b65e4\") " pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"
Oct 31 04:22:56.308270 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.308249    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e83740c-ab5a-49f8-9d97-1fadd09b65e4-plugins-config-volume\") pod \"payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz\" (UID: \"0e83740c-ab5a-49f8-9d97-1fadd09b65e4\") " pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"
Oct 31 04:22:56.312999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.312982    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"
Oct 31 04:22:56.365322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.365286    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cd9r\" (UniqueName: \"kubernetes.io/projected/0e83740c-ab5a-49f8-9d97-1fadd09b65e4-kube-api-access-5cd9r\") pod \"payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz\" (UID: \"0e83740c-ab5a-49f8-9d97-1fadd09b65e4\") " pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"
Oct 31 04:22:56.464943 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.464906    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"
Oct 31 04:22:56.688536 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:22:56.688500    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8722d22a_ef10_4b87_9616_80299e5a6279.slice/crio-0ce1f0c82227e0c4ede0d0c35cea82e0610ac23cdb31cf6a917e2933f8ac3294 WatchSource:0}: Error finding container 0ce1f0c82227e0c4ede0d0c35cea82e0610ac23cdb31cf6a917e2933f8ac3294: Status 404 returned error can't find the container with id 0ce1f0c82227e0c4ede0d0c35cea82e0610ac23cdb31cf6a917e2933f8ac3294
Oct 31 04:22:56.690236 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.690216    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"]
Oct 31 04:22:56.831628 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:22:56.831595    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e83740c_ab5a_49f8_9d97_1fadd09b65e4.slice/crio-2cc573ba4e81bda2ecf2363384ab312a6a3115f1dd9175e426c3e1a127b617a6 WatchSource:0}: Error finding container 2cc573ba4e81bda2ecf2363384ab312a6a3115f1dd9175e426c3e1a127b617a6: Status 404 returned error can't find the container with id 2cc573ba4e81bda2ecf2363384ab312a6a3115f1dd9175e426c3e1a127b617a6
Oct 31 04:22:56.833844 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:56.833807    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"]
Oct 31 04:22:57.239858 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:57.239819    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz" event={"ID":"0e83740c-ab5a-49f8-9d97-1fadd09b65e4","Type":"ContainerStarted","Data":"8c187be809aded781cad1814615bed525b7b76c38b50ca1c1603abfce6a00a3c"}
Oct 31 04:22:57.240064 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:57.239891    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"
Oct 31 04:22:57.240064 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:57.239913    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz" event={"ID":"0e83740c-ab5a-49f8-9d97-1fadd09b65e4","Type":"ContainerStarted","Data":"2cc573ba4e81bda2ecf2363384ab312a6a3115f1dd9175e426c3e1a127b617a6"}
Oct 31 04:22:57.241366 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:57.241342    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr" event={"ID":"8722d22a-ef10-4b87-9616-80299e5a6279","Type":"ContainerStarted","Data":"46bc0609f6c92ac1a2b176306a3789cf43f514c535b429e8d75b5e3cc0892a33"}
Oct 31 04:22:57.241489 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:57.241387    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr" event={"ID":"8722d22a-ef10-4b87-9616-80299e5a6279","Type":"ContainerStarted","Data":"0ce1f0c82227e0c4ede0d0c35cea82e0610ac23cdb31cf6a917e2933f8ac3294"}
Oct 31 04:22:57.241489 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:57.241435    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"
Oct 31 04:22:57.353864 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:57.353778    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz" podStartSLOduration=1.353760942 podStartE2EDuration="1.353760942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:22:57.352998371 +0000 UTC m=+4530.319331716" watchObservedRunningTime="2026-10-31 04:22:57.353760942 +0000 UTC m=+4530.320094291"
Oct 31 04:22:57.467262 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:57.467191    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr" podStartSLOduration=2.467174291 podStartE2EDuration="2.467174291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:22:57.466347163 +0000 UTC m=+4530.432680518" watchObservedRunningTime="2026-10-31 04:22:57.467174291 +0000 UTC m=+4530.433507634"
Oct 31 04:22:57.959014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:57.958985    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:22:58.172934 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:22:58.172901    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:00.009264 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:00.009229    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:23:00.021960 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:00.021936    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:23:00.086139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:00.086096    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9" podUID="25920611-22a3-431c-b9cd-140eddaee0d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:23:00.270929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:00.270855    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:23:03.224293 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:03.224257    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:06.983679 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:23:06.983640    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:23:07.996336 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:07.996307    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:08.223149 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:08.223120    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:08.251559 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:08.251490    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"
Oct 31 04:23:08.251706 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:08.251698    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"
Oct 31 04:23:10.086710 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:10.086669    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9" podUID="25920611-22a3-431c-b9cd-140eddaee0d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:23:12.877733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:12.877698    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:13.078358 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.078296    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"
Oct 31 04:23:13.195428 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.195391    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25920611-22a3-431c-b9cd-140eddaee0d0-kube-api-access-jfv9l\" (UniqueName: \"kubernetes.io/projected/25920611-22a3-431c-b9cd-140eddaee0d0-kube-api-access-jfv9l\") pod \"25920611-22a3-431c-b9cd-140eddaee0d0\" (UID: \"25920611-22a3-431c-b9cd-140eddaee0d0\") "
Oct 31 04:23:13.195607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.195485    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25920611-22a3-431c-b9cd-140eddaee0d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25920611-22a3-431c-b9cd-140eddaee0d0-plugins-config-volume\") pod \"25920611-22a3-431c-b9cd-140eddaee0d0\" (UID: \"25920611-22a3-431c-b9cd-140eddaee0d0\") "
Oct 31 04:23:13.195900 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.195873    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25920611-22a3-431c-b9cd-140eddaee0d0-plugins-config-volume" pod "25920611-22a3-431c-b9cd-140eddaee0d0" (UID: "25920611-22a3-431c-b9cd-140eddaee0d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:23:13.196072 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.196000    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25920611-22a3-431c-b9cd-140eddaee0d0-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:13.197689 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.197663    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25920611-22a3-431c-b9cd-140eddaee0d0-kube-api-access-jfv9l" pod "25920611-22a3-431c-b9cd-140eddaee0d0" (UID: "25920611-22a3-431c-b9cd-140eddaee0d0"). InnerVolumeSpecName "kube-api-access-jfv9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:23:13.297277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.297249    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfv9l\" (UniqueName: \"kubernetes.io/projected/25920611-22a3-431c-b9cd-140eddaee0d0-kube-api-access-jfv9l\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:13.332506 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.332441    4900 generic.go:356] "Generic (PLEG): container finished" podID="25920611-22a3-431c-b9cd-140eddaee0d0" containerID="8f5d6dd003f40db5fe7b43bb53682ed56b125e6820269f7ca824557cc56f73f9" exitCode=0
Oct 31 04:23:13.332639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.332503    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"
Oct 31 04:23:13.332639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.332518    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9" event={"ID":"25920611-22a3-431c-b9cd-140eddaee0d0","Type":"ContainerDied","Data":"8f5d6dd003f40db5fe7b43bb53682ed56b125e6820269f7ca824557cc56f73f9"}
Oct 31 04:23:13.332639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.332580    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9" event={"ID":"25920611-22a3-431c-b9cd-140eddaee0d0","Type":"ContainerDied","Data":"d5d8919cee580c756bd48ff9ded609e8e7436c6e1201ed8885f691823cbd0997"}
Oct 31 04:23:13.332639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.332611    4900 scope.go:122] "RemoveContainer" containerID="8f5d6dd003f40db5fe7b43bb53682ed56b125e6820269f7ca824557cc56f73f9"
Oct 31 04:23:13.342756 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.342722    4900 scope.go:122] "RemoveContainer" containerID="8f5d6dd003f40db5fe7b43bb53682ed56b125e6820269f7ca824557cc56f73f9"
Oct 31 04:23:13.343066 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:23:13.343040    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5d6dd003f40db5fe7b43bb53682ed56b125e6820269f7ca824557cc56f73f9\": container with ID starting with 8f5d6dd003f40db5fe7b43bb53682ed56b125e6820269f7ca824557cc56f73f9 not found: ID does not exist" containerID="8f5d6dd003f40db5fe7b43bb53682ed56b125e6820269f7ca824557cc56f73f9"
Oct 31 04:23:13.343148 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.343078    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5d6dd003f40db5fe7b43bb53682ed56b125e6820269f7ca824557cc56f73f9"} err="failed to get container status \"8f5d6dd003f40db5fe7b43bb53682ed56b125e6820269f7ca824557cc56f73f9\": rpc error: code = NotFound desc = could not find container \"8f5d6dd003f40db5fe7b43bb53682ed56b125e6820269f7ca824557cc56f73f9\": container with ID starting with 8f5d6dd003f40db5fe7b43bb53682ed56b125e6820269f7ca824557cc56f73f9 not found: ID does not exist"
Oct 31 04:23:13.497324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.497298    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"]
Oct 31 04:23:13.501652 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:13.501626    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-dd5b85-68fc6966f9-5msv9"]
Oct 31 04:23:15.153925 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:15.153891    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25920611-22a3-431c-b9cd-140eddaee0d0" path="/var/lib/kubelet/pods/25920611-22a3-431c-b9cd-140eddaee0d0/volumes"
Oct 31 04:23:16.693086 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:16.693044    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:16.931182 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:16.931141    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:17.494617 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:17.494574    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:17.788897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:17.788808    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:22.694524 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:22.694493    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:23.083268 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:23.083237    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:27.203437 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.203404    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"]
Oct 31 04:23:27.203973 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.203957    4900 cpu_manager.go:414] "containerMap: removing container" podUID="25920611-22a3-431c-b9cd-140eddaee0d0" containerName="payload-pre-processing"
Oct 31 04:23:27.204032 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.203973    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25920611-22a3-431c-b9cd-140eddaee0d0" containerName="payload-pre-processing"
Oct 31 04:23:27.204092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.204083    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="25920611-22a3-431c-b9cd-140eddaee0d0" containerName="payload-pre-processing"
Oct 31 04:23:27.208733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.208716    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.254255 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.254219    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.254418 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.254278    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.254418 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.254384    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa31682-6c46-423a-a806-37fbf83916e5-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.254499 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.254416    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xzv4\" (UniqueName: \"kubernetes.io/projected/dfa31682-6c46-423a-a806-37fbf83916e5-kube-api-access-4xzv4\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.254537 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.254504    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa31682-6c46-423a-a806-37fbf83916e5-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.254605 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.254591    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa31682-6c46-423a-a806-37fbf83916e5-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.254650 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.254627    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.254752 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.254736    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.254828 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.254783    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.314060 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.314028    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"]
Oct 31 04:23:27.314354 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.314336    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-dd5b85-openshift-default-dockercfg-b24df\""
Oct 31 04:23:27.355710 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.355677    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa31682-6c46-423a-a806-37fbf83916e5-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.355941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.355719    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.355941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.355781    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.355941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.355854    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.355941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.355912    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.356186 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.355965    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.356186 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.356010    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa31682-6c46-423a-a806-37fbf83916e5-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.356186 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.356044    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xzv4\" (UniqueName: \"kubernetes.io/projected/dfa31682-6c46-423a-a806-37fbf83916e5-kube-api-access-4xzv4\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.356186 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.356104    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa31682-6c46-423a-a806-37fbf83916e5-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.356405 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.356265    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-workload-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.356405 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.356290    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-credential-socket\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.356405 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.356383    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-workload-certs\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.356604 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.356581    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-istio-data\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.356853 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.356832    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa31682-6c46-423a-a806-37fbf83916e5-istiod-ca-cert\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.358043 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.358024    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-istio-envoy\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.358273 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.358254    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa31682-6c46-423a-a806-37fbf83916e5-istio-podinfo\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.413050 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.413015    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa31682-6c46-423a-a806-37fbf83916e5-istio-token\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.413211 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.413175    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xzv4\" (UniqueName: \"kubernetes.io/projected/dfa31682-6c46-423a-a806-37fbf83916e5-kube-api-access-4xzv4\") pod \"e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") " pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.488291 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.488257    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:27.522487 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.522463    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:27.940548 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.940474    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"]
Oct 31 04:23:27.973630 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:23:27.973604    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa31682_6c46_423a_a806_37fbf83916e5.slice/crio-c44469af6c038e38117bef5539c31228faf44b9a9bcfbd003468a3b435f6d663 WatchSource:0}: Error finding container c44469af6c038e38117bef5539c31228faf44b9a9bcfbd003468a3b435f6d663: Status 404 returned error can't find the container with id c44469af6c038e38117bef5539c31228faf44b9a9bcfbd003468a3b435f6d663
Oct 31 04:23:27.975985 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.975954    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:23:27.976065 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.976022    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:23:27.976065 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:27.976048    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:23:28.436193 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:28.436153    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb" event={"ID":"dfa31682-6c46-423a-a806-37fbf83916e5","Type":"ContainerStarted","Data":"b59972c2cc1f169cd2ee6b0c9751990c6e23552d680d77f5e2692f5cea8d68d4"}
Oct 31 04:23:28.436709 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:28.436248    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb" event={"ID":"dfa31682-6c46-423a-a806-37fbf83916e5","Type":"ContainerStarted","Data":"c44469af6c038e38117bef5539c31228faf44b9a9bcfbd003468a3b435f6d663"}
Oct 31 04:23:28.523065 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:28.523029    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:28.558810 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:28.557909    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb" podStartSLOduration=1.557888186 podStartE2EDuration="1.557888186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:23:28.552139705 +0000 UTC m=+4561.518473049" watchObservedRunningTime="2026-10-31 04:23:28.557888186 +0000 UTC m=+4561.524221534"
Oct 31 04:23:29.525271 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:29.525231    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 31 04:23:29.525667 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:29.525287    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb" podUID="dfa31682-6c46-423a-a806-37fbf83916e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 31 04:23:30.523563 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:30.523535    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 31 04:23:30.523744 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:30.523580    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb" podUID="dfa31682-6c46-423a-a806-37fbf83916e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 31 04:23:31.453812 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.453426    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"]
Oct 31 04:23:31.453812 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.453693    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x" podUID="8c7e225e-0ba9-499d-9372-7bd7527354c7" containerName="maas-api" containerID="cri-o://f807def13f5fe8e958468f27d9dcebc9293fc593887d726da188537219e6384d" gracePeriod=30
Oct 31 04:23:31.523335 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.523298    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 31 04:23:31.523488 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.523357    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb" podUID="dfa31682-6c46-423a-a806-37fbf83916e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 31 04:23:31.657819 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.652943    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:31.768685 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.768648    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 04:23:31.806575 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.806547    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-metrics-certs\") pod \"8c7e225e-0ba9-499d-9372-7bd7527354c7\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") "
Oct 31 04:23:31.806575 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.806579    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-maas-api-tls\") pod \"8c7e225e-0ba9-499d-9372-7bd7527354c7\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") "
Oct 31 04:23:31.806755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.806629    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c7e225e-0ba9-499d-9372-7bd7527354c7-kube-api-access-7lclh\" (UniqueName: \"kubernetes.io/projected/8c7e225e-0ba9-499d-9372-7bd7527354c7-kube-api-access-7lclh\") pod \"8c7e225e-0ba9-499d-9372-7bd7527354c7\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") "
Oct 31 04:23:31.806755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.806727    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c7e225e-0ba9-499d-9372-7bd7527354c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e225e-0ba9-499d-9372-7bd7527354c7-openshift-service-ca\") pod \"8c7e225e-0ba9-499d-9372-7bd7527354c7\" (UID: \"8c7e225e-0ba9-499d-9372-7bd7527354c7\") "
Oct 31 04:23:31.807158 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.807130    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c7e225e-0ba9-499d-9372-7bd7527354c7-openshift-service-ca" pod "8c7e225e-0ba9-499d-9372-7bd7527354c7" (UID: "8c7e225e-0ba9-499d-9372-7bd7527354c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:23:31.807277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.807261    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c7e225e-0ba9-499d-9372-7bd7527354c7-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:31.808751 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.808727    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-maas-api-tls" pod "8c7e225e-0ba9-499d-9372-7bd7527354c7" (UID: "8c7e225e-0ba9-499d-9372-7bd7527354c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:23:31.808850 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.808751    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-metrics-certs" pod "8c7e225e-0ba9-499d-9372-7bd7527354c7" (UID: "8c7e225e-0ba9-499d-9372-7bd7527354c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:23:31.808850 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.808756    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c7e225e-0ba9-499d-9372-7bd7527354c7-kube-api-access-7lclh" pod "8c7e225e-0ba9-499d-9372-7bd7527354c7" (UID: "8c7e225e-0ba9-499d-9372-7bd7527354c7"). InnerVolumeSpecName "kube-api-access-7lclh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:23:31.908744 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.908706    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:31.908744 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.908739    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c7e225e-0ba9-499d-9372-7bd7527354c7-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:31.908970 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:31.908751    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lclh\" (UniqueName: \"kubernetes.io/projected/8c7e225e-0ba9-499d-9372-7bd7527354c7-kube-api-access-7lclh\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:32.483505 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:32.483466    4900 generic.go:356] "Generic (PLEG): container finished" podID="8c7e225e-0ba9-499d-9372-7bd7527354c7" containerID="f807def13f5fe8e958468f27d9dcebc9293fc593887d726da188537219e6384d" exitCode=0
Oct 31 04:23:32.483970 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:32.483516    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"
Oct 31 04:23:32.483970 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:32.483551    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x" event={"ID":"8c7e225e-0ba9-499d-9372-7bd7527354c7","Type":"ContainerDied","Data":"f807def13f5fe8e958468f27d9dcebc9293fc593887d726da188537219e6384d"}
Oct 31 04:23:32.483970 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:32.483602    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x" event={"ID":"8c7e225e-0ba9-499d-9372-7bd7527354c7","Type":"ContainerDied","Data":"35d708329f9cc9d21a1b23fa56687a3c97922b9f29c0db1ae653ad0d5c2bd122"}
Oct 31 04:23:32.483970 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:32.483630    4900 scope.go:122] "RemoveContainer" containerID="f807def13f5fe8e958468f27d9dcebc9293fc593887d726da188537219e6384d"
Oct 31 04:23:32.494063 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:32.494041    4900 scope.go:122] "RemoveContainer" containerID="f807def13f5fe8e958468f27d9dcebc9293fc593887d726da188537219e6384d"
Oct 31 04:23:32.494314 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:23:32.494291    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f807def13f5fe8e958468f27d9dcebc9293fc593887d726da188537219e6384d\": container with ID starting with f807def13f5fe8e958468f27d9dcebc9293fc593887d726da188537219e6384d not found: ID does not exist" containerID="f807def13f5fe8e958468f27d9dcebc9293fc593887d726da188537219e6384d"
Oct 31 04:23:32.494421 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:32.494321    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f807def13f5fe8e958468f27d9dcebc9293fc593887d726da188537219e6384d"} err="failed to get container status \"f807def13f5fe8e958468f27d9dcebc9293fc593887d726da188537219e6384d\": rpc error: code = NotFound desc = could not find container \"f807def13f5fe8e958468f27d9dcebc9293fc593887d726da188537219e6384d\": container with ID starting with f807def13f5fe8e958468f27d9dcebc9293fc593887d726da188537219e6384d not found: ID does not exist"
Oct 31 04:23:32.523602 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:32.523580    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 31 04:23:32.523720 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:32.523630    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb" podUID="dfa31682-6c46-423a-a806-37fbf83916e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 31 04:23:32.662244 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:32.662209    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"]
Oct 31 04:23:32.664699 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:32.664675    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-dd5b85-7c8dcb568d-b6r5x"]
Oct 31 04:23:33.167339 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:33.167305    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c7e225e-0ba9-499d-9372-7bd7527354c7" path="/var/lib/kubelet/pods/8c7e225e-0ba9-499d-9372-7bd7527354c7/volumes"
Oct 31 04:23:33.397999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:33.397962    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:33.523381 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:33.523349    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 31 04:23:33.523818 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:33.523408    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb" podUID="dfa31682-6c46-423a-a806-37fbf83916e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 31 04:23:34.522903 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:34.522868    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body=
Oct 31 04:23:34.523076 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:34.522929    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb" podUID="dfa31682-6c46-423a-a806-37fbf83916e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused"
Oct 31 04:23:36.551107 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:36.551055    4900 patch_prober.go:29] interesting pod/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:23:36.551543 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:36.551140    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb" podUID="dfa31682-6c46-423a-a806-37fbf83916e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:23:36.555614 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:36.555586    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:36.556045 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:36.556016    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:36.557132 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:36.557114    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:36.983970 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:23:36.983933    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:23:37.759751 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:37.759713    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:37.980692 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:37.980646    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:38.227304 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:38.227214    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:39.102176 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:39.102141    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:39.319249 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:39.319215    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:40.690063 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:40.690013    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:41.083477 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:41.083448    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"]
Oct 31 04:23:41.083661 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:41.083633    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb" podUID="dfa31682-6c46-423a-a806-37fbf83916e5" containerName="istio-proxy" containerID="cri-o://b59972c2cc1f169cd2ee6b0c9751990c6e23552d680d77f5e2692f5cea8d68d4" gracePeriod=30
Oct 31 04:23:41.194757 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:41.194724    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:43.318025 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:43.317989    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:43.534958 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:43.534926    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:46.356355 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.356298    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:46.484112 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.484081    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-istio-envoy\") pod \"dfa31682-6c46-423a-a806-37fbf83916e5\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") "
Oct 31 04:23:46.484291 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.484136    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa31682-6c46-423a-a806-37fbf83916e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa31682-6c46-423a-a806-37fbf83916e5-istiod-ca-cert\") pod \"dfa31682-6c46-423a-a806-37fbf83916e5\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") "
Oct 31 04:23:46.484291 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.484189    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa31682-6c46-423a-a806-37fbf83916e5-kube-api-access-4xzv4\" (UniqueName: \"kubernetes.io/projected/dfa31682-6c46-423a-a806-37fbf83916e5-kube-api-access-4xzv4\") pod \"dfa31682-6c46-423a-a806-37fbf83916e5\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") "
Oct 31 04:23:46.484291 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.484241    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-workload-certs\") pod \"dfa31682-6c46-423a-a806-37fbf83916e5\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") "
Oct 31 04:23:46.484461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.484292    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa31682-6c46-423a-a806-37fbf83916e5-istio-token\" (UniqueName: \"kubernetes.io/projected/dfa31682-6c46-423a-a806-37fbf83916e5-istio-token\") pod \"dfa31682-6c46-423a-a806-37fbf83916e5\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") "
Oct 31 04:23:46.484461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.484348    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfa31682-6c46-423a-a806-37fbf83916e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa31682-6c46-423a-a806-37fbf83916e5-istio-podinfo\") pod \"dfa31682-6c46-423a-a806-37fbf83916e5\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") "
Oct 31 04:23:46.484461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.484391    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-workload-socket\") pod \"dfa31682-6c46-423a-a806-37fbf83916e5\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") "
Oct 31 04:23:46.484461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.484447    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-istio-data\") pod \"dfa31682-6c46-423a-a806-37fbf83916e5\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") "
Oct 31 04:23:46.484704 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.484477    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-credential-socket\") pod \"dfa31682-6c46-423a-a806-37fbf83916e5\" (UID: \"dfa31682-6c46-423a-a806-37fbf83916e5\") "
Oct 31 04:23:46.484704 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.484586    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-workload-certs" pod "dfa31682-6c46-423a-a806-37fbf83916e5" (UID: "dfa31682-6c46-423a-a806-37fbf83916e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:23:46.484704 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.484688    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-workload-socket" pod "dfa31682-6c46-423a-a806-37fbf83916e5" (UID: "dfa31682-6c46-423a-a806-37fbf83916e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:23:46.484929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.484849    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-istio-data" pod "dfa31682-6c46-423a-a806-37fbf83916e5" (UID: "dfa31682-6c46-423a-a806-37fbf83916e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:23:46.484929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.484915    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-credential-socket" pod "dfa31682-6c46-423a-a806-37fbf83916e5" (UID: "dfa31682-6c46-423a-a806-37fbf83916e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:23:46.485160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.485096    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.485160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.485126    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.485160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.485141    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.485160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.485143    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa31682-6c46-423a-a806-37fbf83916e5-istiod-ca-cert" pod "dfa31682-6c46-423a-a806-37fbf83916e5" (UID: "dfa31682-6c46-423a-a806-37fbf83916e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:23:46.485160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.485156    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.486530 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.486505    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-istio-envoy" pod "dfa31682-6c46-423a-a806-37fbf83916e5" (UID: "dfa31682-6c46-423a-a806-37fbf83916e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:23:46.486631 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.486567    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa31682-6c46-423a-a806-37fbf83916e5-kube-api-access-4xzv4" pod "dfa31682-6c46-423a-a806-37fbf83916e5" (UID: "dfa31682-6c46-423a-a806-37fbf83916e5"). InnerVolumeSpecName "kube-api-access-4xzv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:23:46.486631 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.486589    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa31682-6c46-423a-a806-37fbf83916e5-istio-token" pod "dfa31682-6c46-423a-a806-37fbf83916e5" (UID: "dfa31682-6c46-423a-a806-37fbf83916e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:23:46.486631 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.486625    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfa31682-6c46-423a-a806-37fbf83916e5-istio-podinfo" pod "dfa31682-6c46-423a-a806-37fbf83916e5" (UID: "dfa31682-6c46-423a-a806-37fbf83916e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:23:46.560475 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.560439    4900 generic.go:356] "Generic (PLEG): container finished" podID="dfa31682-6c46-423a-a806-37fbf83916e5" containerID="b59972c2cc1f169cd2ee6b0c9751990c6e23552d680d77f5e2692f5cea8d68d4" exitCode=0
Oct 31 04:23:46.560659 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.560492    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"
Oct 31 04:23:46.560659 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.560527    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb" event={"ID":"dfa31682-6c46-423a-a806-37fbf83916e5","Type":"ContainerDied","Data":"b59972c2cc1f169cd2ee6b0c9751990c6e23552d680d77f5e2692f5cea8d68d4"}
Oct 31 04:23:46.560659 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.560589    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb" event={"ID":"dfa31682-6c46-423a-a806-37fbf83916e5","Type":"ContainerDied","Data":"c44469af6c038e38117bef5539c31228faf44b9a9bcfbd003468a3b435f6d663"}
Oct 31 04:23:46.560659 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.560613    4900 scope.go:122] "RemoveContainer" containerID="b59972c2cc1f169cd2ee6b0c9751990c6e23552d680d77f5e2692f5cea8d68d4"
Oct 31 04:23:46.574090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.574069    4900 scope.go:122] "RemoveContainer" containerID="b59972c2cc1f169cd2ee6b0c9751990c6e23552d680d77f5e2692f5cea8d68d4"
Oct 31 04:23:46.574417 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:23:46.574397    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59972c2cc1f169cd2ee6b0c9751990c6e23552d680d77f5e2692f5cea8d68d4\": container with ID starting with b59972c2cc1f169cd2ee6b0c9751990c6e23552d680d77f5e2692f5cea8d68d4 not found: ID does not exist" containerID="b59972c2cc1f169cd2ee6b0c9751990c6e23552d680d77f5e2692f5cea8d68d4"
Oct 31 04:23:46.574567 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.574426    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59972c2cc1f169cd2ee6b0c9751990c6e23552d680d77f5e2692f5cea8d68d4"} err="failed to get container status \"b59972c2cc1f169cd2ee6b0c9751990c6e23552d680d77f5e2692f5cea8d68d4\": rpc error: code = NotFound desc = could not find container \"b59972c2cc1f169cd2ee6b0c9751990c6e23552d680d77f5e2692f5cea8d68d4\": container with ID starting with b59972c2cc1f169cd2ee6b0c9751990c6e23552d680d77f5e2692f5cea8d68d4 not found: ID does not exist"
Oct 31 04:23:46.586593 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.586568    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa31682-6c46-423a-a806-37fbf83916e5-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.586593 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.586592    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa31682-6c46-423a-a806-37fbf83916e5-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.586768 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.586602    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa31682-6c46-423a-a806-37fbf83916e5-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.586768 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.586613    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa31682-6c46-423a-a806-37fbf83916e5-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.586768 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.586621    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xzv4\" (UniqueName: \"kubernetes.io/projected/dfa31682-6c46-423a-a806-37fbf83916e5-kube-api-access-4xzv4\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:23:46.726161 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.726071    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"]
Oct 31 04:23:46.731434 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:46.731406    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-dd5b85-openshift-default-df7b5cdd7-whgrb"]
Oct 31 04:23:47.153906 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:47.153871    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa31682-6c46-423a-a806-37fbf83916e5" path="/var/lib/kubelet/pods/dfa31682-6c46-423a-a806-37fbf83916e5/volumes"
Oct 31 04:23:48.451372 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:48.451337    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:48.673919 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:48.673888    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:53.200174 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:53.200140    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:54.481437 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:54.481391    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:23:57.980534 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:23:57.980494    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:02.773119 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:02.773087    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:06.984921 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:24:06.984882    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:24:07.552811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:07.545311    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:08.402183 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:08.402140    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:12.266271 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:12.266188    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:15.723823 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:15.723712    4900 generic.go:356] "Generic (PLEG): container finished" podID="36e17b74-45d2-4d36-842f-1c1868c501e0" containerID="5f2f03666608e01fe6c38cf054d95fa8019df4b1e303defca2b7bf9ab7911e83" exitCode=0
Oct 31 04:24:15.723823 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:15.723804    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj" event={"ID":"36e17b74-45d2-4d36-842f-1c1868c501e0","Type":"ContainerDied","Data":"5f2f03666608e01fe6c38cf054d95fa8019df4b1e303defca2b7bf9ab7911e83"}
Oct 31 04:24:15.724261 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:15.724198    4900 scope.go:122] "RemoveContainer" containerID="5f2f03666608e01fe6c38cf054d95fa8019df4b1e303defca2b7bf9ab7911e83"
Oct 31 04:24:15.725600 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:15.725575    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:24:15.725651 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:15.725642    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:24:15.725687 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:15.725673    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:24:16.731193 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:16.731153    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj" event={"ID":"36e17b74-45d2-4d36-842f-1c1868c501e0","Type":"ContainerStarted","Data":"30c186b1f373172b66a294b7ab7eeab8078cb5f5072ef4427ee129d396c24fac"}
Oct 31 04:24:17.025151 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.025122    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:17.121507 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.121472    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"]
Oct 31 04:24:17.122066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.122044    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8c7e225e-0ba9-499d-9372-7bd7527354c7" containerName="maas-api"
Oct 31 04:24:17.122066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.122064    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c7e225e-0ba9-499d-9372-7bd7527354c7" containerName="maas-api"
Oct 31 04:24:17.122316 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.122099    4900 cpu_manager.go:414] "containerMap: removing container" podUID="dfa31682-6c46-423a-a806-37fbf83916e5" containerName="istio-proxy"
Oct 31 04:24:17.122316 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.122107    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa31682-6c46-423a-a806-37fbf83916e5" containerName="istio-proxy"
Oct 31 04:24:17.122316 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.122274    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c7e225e-0ba9-499d-9372-7bd7527354c7" containerName="maas-api"
Oct 31 04:24:17.122316 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.122287    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa31682-6c46-423a-a806-37fbf83916e5" containerName="istio-proxy"
Oct 31 04:24:17.165592 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.165547    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-tmp-dir\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.165840 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.165602    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnxgs\" (UniqueName: \"kubernetes.io/projected/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-kube-api-access-qnxgs\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.165840 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.165645    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-model-cache\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.165840 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.165732    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-home\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.166047 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.165889    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-dshm\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.166047 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.165933    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-tls-certs\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.186060 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.186032    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.193924 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.193891    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"]
Oct 31 04:24:17.275954 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.274491    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-502ec2\"/\"rate-test-model-502ec2-kserve-self-signed-certs\""
Oct 31 04:24:17.275954 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.274665    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-502ec2\"/\"kube-root-ca.crt\""
Oct 31 04:24:17.275954 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.275613    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-502ec2\"/\"openshift-service-ca.crt\""
Oct 31 04:24:17.275954 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.275726    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-502ec2\"/\"default-dockercfg-vk2p6\""
Oct 31 04:24:17.276277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.276027    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-dshm\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.276277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.276061    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-tls-certs\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.276277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.276099    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-tmp-dir\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.276277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.276115    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnxgs\" (UniqueName: \"kubernetes.io/projected/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-kube-api-access-qnxgs\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.276277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.276139    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-model-cache\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.276277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.276200    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-home\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.276920 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.276883    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-model-cache\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.277112 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.277093    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-home\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.277320 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.277302    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-tmp-dir\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.287914 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.286987    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-dshm\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.287914 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.287245    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-tls-certs\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.288165 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.288137    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:17.366497 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.366458    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnxgs\" (UniqueName: \"kubernetes.io/projected/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-kube-api-access-qnxgs\") pod \"rate-test-model-502ec2-kserve-7d9847c485-8f29t\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") " pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.503348 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.503312    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:17.655054 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.640098    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:24:17.659593 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.659560    4900 patch_prober.go:29] interesting pod/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 31 04:24:17.659727 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.659616    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj" podUID="36e17b74-45d2-4d36-842f-1c1868c501e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 31 04:24:17.920919 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:24:17.920890    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48a65552_b7f7_421c_b714_3e3f2f2b3dc2.slice/crio-2df604d1c093ba16c1d6ebdc2b1d08ecd765a2ff0c87148005008ddfdd2a1756 WatchSource:0}: Error finding container 2df604d1c093ba16c1d6ebdc2b1d08ecd765a2ff0c87148005008ddfdd2a1756: Status 404 returned error can't find the container with id 2df604d1c093ba16c1d6ebdc2b1d08ecd765a2ff0c87148005008ddfdd2a1756
Oct 31 04:24:17.923511 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:17.923415    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"]
Oct 31 04:24:18.650810 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:18.638141    4900 patch_prober.go:29] interesting pod/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 31 04:24:18.650810 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:18.638203    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj" podUID="36e17b74-45d2-4d36-842f-1c1868c501e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 31 04:24:18.768975 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:18.768941    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t" event={"ID":"48a65552-b7f7-421c-b714-3e3f2f2b3dc2","Type":"ContainerStarted","Data":"d5d5ed9e778845eb0c008f6ff797dd7a6f75f8a16b9e1eb240a1fb526138b598"}
Oct 31 04:24:18.768975 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:18.768980    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t" event={"ID":"48a65552-b7f7-421c-b714-3e3f2f2b3dc2","Type":"ContainerStarted","Data":"2df604d1c093ba16c1d6ebdc2b1d08ecd765a2ff0c87148005008ddfdd2a1756"}
Oct 31 04:24:18.884176 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:18.884125    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t" podStartSLOduration=1.8841102140000001 podStartE2EDuration="1.884110214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:24:18.882831429 +0000 UTC m=+4611.849164770" watchObservedRunningTime="2026-10-31 04:24:18.884110214 +0000 UTC m=+4611.850443557"
Oct 31 04:24:19.504296 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:19.504255    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:19.651827 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:19.645177    4900 patch_prober.go:29] interesting pod/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 31 04:24:19.651827 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:19.645239    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj" podUID="36e17b74-45d2-4d36-842f-1c1868c501e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 31 04:24:20.658925 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:20.658886    4900 patch_prober.go:29] interesting pod/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 31 04:24:20.659354 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:20.658942    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj" podUID="36e17b74-45d2-4d36-842f-1c1868c501e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 31 04:24:21.206369 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:21.206339    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:24:21.659058 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:21.656000    4900 patch_prober.go:29] interesting pod/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body=
Oct 31 04:24:21.659058 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:21.656071    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj" podUID="36e17b74-45d2-4d36-842f-1c1868c501e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused"
Oct 31 04:24:21.788989 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:21.788952    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:22.641380 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:22.641348    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:24:22.642323 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:22.642299    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:24:26.600165 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:26.600133    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:27.503574 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:27.503539    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:27.516199 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:27.516170    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:27.836092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:27.836019    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:24:31.514917 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:31.514886    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:36.304473 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:36.304440    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:36.984236 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:24:36.984196    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:24:41.184931 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:41.184901    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:44.627014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:44.626973    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:45.521038 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:45.521002    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:45.769861 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:45.769827    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:46.005769 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:46.005733    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:46.444315 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:46.444225    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:51.190339 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:51.190297    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:52.370258 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:52.370174    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:24:56.013350 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:24:56.013316    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:00.767043 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:00.767009    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:05.511690 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:05.511652    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:06.984720 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:25:06.984680    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:25:08.727116 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:08.727069    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:10.390195 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:10.390163    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:13.612623 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:13.612585    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:15.218894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:15.218854    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:19.205923 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:19.205885    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:19.964388 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:19.964354    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:24.212539 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:24.212505    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:24.818008 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:24.817975    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:25.042164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:25.042129    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:25.978090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:25.978054    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"]
Oct 31 04:25:26.451943 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:26.451910    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:26.730426 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:26.730340    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:29.901197 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:29.901158    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:30.691518 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:30.691488    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:30.978827 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:30.978691    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp" podUID="ca32c87f-ca00-4404-9560-22463f06c330" containerName="payload-processing" containerID="cri-o://3d75725344d0bbb75171871e3186f01aea2e1fcfc31a51ccb4fb0fe00477a69d" gracePeriod=25
Oct 31 04:25:31.911025 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:31.910981    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp" podUID="ca32c87f-ca00-4404-9560-22463f06c330" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:25:34.703681 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:34.703648    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:36.984583 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:25:36.984547    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:25:39.493028 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:39.492951    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:41.911001 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:41.910961    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp" podUID="ca32c87f-ca00-4404-9560-22463f06c330" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:25:44.220049 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:44.220016    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:48.996192 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:48.996152    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:51.237878 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.237814    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"
Oct 31 04:25:51.306645 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.306606    4900 generic.go:356] "Generic (PLEG): container finished" podID="ca32c87f-ca00-4404-9560-22463f06c330" containerID="3d75725344d0bbb75171871e3186f01aea2e1fcfc31a51ccb4fb0fe00477a69d" exitCode=0
Oct 31 04:25:51.306874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.306653    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp" event={"ID":"ca32c87f-ca00-4404-9560-22463f06c330","Type":"ContainerDied","Data":"3d75725344d0bbb75171871e3186f01aea2e1fcfc31a51ccb4fb0fe00477a69d"}
Oct 31 04:25:51.306874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.306683    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"
Oct 31 04:25:51.306874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.306733    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp" event={"ID":"ca32c87f-ca00-4404-9560-22463f06c330","Type":"ContainerDied","Data":"fb55c935a5562a3f15675c672b198811eac98df42be92266171365c4aa44204a"}
Oct 31 04:25:51.306874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.306765    4900 scope.go:122] "RemoveContainer" containerID="3d75725344d0bbb75171871e3186f01aea2e1fcfc31a51ccb4fb0fe00477a69d"
Oct 31 04:25:51.312485 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.312459    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca32c87f-ca00-4404-9560-22463f06c330-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca32c87f-ca00-4404-9560-22463f06c330-plugins-config-volume\") pod \"ca32c87f-ca00-4404-9560-22463f06c330\" (UID: \"ca32c87f-ca00-4404-9560-22463f06c330\") "
Oct 31 04:25:51.312654 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.312635    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca32c87f-ca00-4404-9560-22463f06c330-kube-api-access-k84xx\" (UniqueName: \"kubernetes.io/projected/ca32c87f-ca00-4404-9560-22463f06c330-kube-api-access-k84xx\") pod \"ca32c87f-ca00-4404-9560-22463f06c330\" (UID: \"ca32c87f-ca00-4404-9560-22463f06c330\") "
Oct 31 04:25:51.312909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.312882    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca32c87f-ca00-4404-9560-22463f06c330-plugins-config-volume" pod "ca32c87f-ca00-4404-9560-22463f06c330" (UID: "ca32c87f-ca00-4404-9560-22463f06c330"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:25:51.313094 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.313077    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca32c87f-ca00-4404-9560-22463f06c330-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:25:51.318118 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.318088    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca32c87f-ca00-4404-9560-22463f06c330-kube-api-access-k84xx" pod "ca32c87f-ca00-4404-9560-22463f06c330" (UID: "ca32c87f-ca00-4404-9560-22463f06c330"). InnerVolumeSpecName "kube-api-access-k84xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:25:51.318824 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.318779    4900 scope.go:122] "RemoveContainer" containerID="3d75725344d0bbb75171871e3186f01aea2e1fcfc31a51ccb4fb0fe00477a69d"
Oct 31 04:25:51.319182 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:25:51.319152    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d75725344d0bbb75171871e3186f01aea2e1fcfc31a51ccb4fb0fe00477a69d\": container with ID starting with 3d75725344d0bbb75171871e3186f01aea2e1fcfc31a51ccb4fb0fe00477a69d not found: ID does not exist" containerID="3d75725344d0bbb75171871e3186f01aea2e1fcfc31a51ccb4fb0fe00477a69d"
Oct 31 04:25:51.319281 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.319193    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d75725344d0bbb75171871e3186f01aea2e1fcfc31a51ccb4fb0fe00477a69d"} err="failed to get container status \"3d75725344d0bbb75171871e3186f01aea2e1fcfc31a51ccb4fb0fe00477a69d\": rpc error: code = NotFound desc = could not find container \"3d75725344d0bbb75171871e3186f01aea2e1fcfc31a51ccb4fb0fe00477a69d\": container with ID starting with 3d75725344d0bbb75171871e3186f01aea2e1fcfc31a51ccb4fb0fe00477a69d not found: ID does not exist"
Oct 31 04:25:51.414343 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.414308    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k84xx\" (UniqueName: \"kubernetes.io/projected/ca32c87f-ca00-4404-9560-22463f06c330-kube-api-access-k84xx\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:25:51.729031 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.728995    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"]
Oct 31 04:25:51.731612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:51.731586    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-861dfd-655b9bdd89-74pgp"]
Oct 31 04:25:53.152613 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:53.152573    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca32c87f-ca00-4404-9560-22463f06c330" path="/var/lib/kubelet/pods/ca32c87f-ca00-4404-9560-22463f06c330/volumes"
Oct 31 04:25:53.178907 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:53.178876    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:54.980521 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:54.980491    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:25:59.789113 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:25:59.789072    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:00.613937 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.613901    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"]
Oct 31 04:26:00.614410 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.614397    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ca32c87f-ca00-4404-9560-22463f06c330" containerName="payload-processing"
Oct 31 04:26:00.614456 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.614411    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca32c87f-ca00-4404-9560-22463f06c330" containerName="payload-processing"
Oct 31 04:26:00.614532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.614522    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca32c87f-ca00-4404-9560-22463f06c330" containerName="payload-processing"
Oct 31 04:26:00.619306 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.619291    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"
Oct 31 04:26:00.624097 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.624077    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26655c2d-eaf1-482b-901f-8c52ec472f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-646f8765-qwqxb\" (UID: \"26655c2d-eaf1-482b-901f-8c52ec472f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"
Oct 31 04:26:00.624193 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.624123    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26655c2d-eaf1-482b-901f-8c52ec472f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-646f8765-qwqxb\" (UID: \"26655c2d-eaf1-482b-901f-8c52ec472f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"
Oct 31 04:26:00.624234 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.624201    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rntk\" (UniqueName: \"kubernetes.io/projected/26655c2d-eaf1-482b-901f-8c52ec472f1d-kube-api-access-5rntk\") pod \"maas-api-revoke-sub-646f8765-qwqxb\" (UID: \"26655c2d-eaf1-482b-901f-8c52ec472f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"
Oct 31 04:26:00.670076 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.670043    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"]
Oct 31 04:26:00.671316 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.671294    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:26:00.717512 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.717485    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:26:00.725847 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.725818    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26655c2d-eaf1-482b-901f-8c52ec472f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-646f8765-qwqxb\" (UID: \"26655c2d-eaf1-482b-901f-8c52ec472f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"
Oct 31 04:26:00.725994 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.725896    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26655c2d-eaf1-482b-901f-8c52ec472f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-646f8765-qwqxb\" (UID: \"26655c2d-eaf1-482b-901f-8c52ec472f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"
Oct 31 04:26:00.725994 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.725964    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rntk\" (UniqueName: \"kubernetes.io/projected/26655c2d-eaf1-482b-901f-8c52ec472f1d-kube-api-access-5rntk\") pod \"maas-api-revoke-sub-646f8765-qwqxb\" (UID: \"26655c2d-eaf1-482b-901f-8c52ec472f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"
Oct 31 04:26:00.727718 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.727690    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26655c2d-eaf1-482b-901f-8c52ec472f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-646f8765-qwqxb\" (UID: \"26655c2d-eaf1-482b-901f-8c52ec472f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"
Oct 31 04:26:00.737497 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.737470    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26655c2d-eaf1-482b-901f-8c52ec472f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-646f8765-qwqxb\" (UID: \"26655c2d-eaf1-482b-901f-8c52ec472f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"
Oct 31 04:26:00.782783 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.782752    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rntk\" (UniqueName: \"kubernetes.io/projected/26655c2d-eaf1-482b-901f-8c52ec472f1d-kube-api-access-5rntk\") pod \"maas-api-revoke-sub-646f8765-qwqxb\" (UID: \"26655c2d-eaf1-482b-901f-8c52ec472f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"
Oct 31 04:26:00.933044 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:00.932954    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"
Oct 31 04:26:01.300006 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:26:01.299974    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26655c2d_eaf1_482b_901f_8c52ec472f1d.slice/crio-8d6d7177b05683af7d811ca74912b36fdd98237bb094766c606fe1618eee805a WatchSource:0}: Error finding container 8d6d7177b05683af7d811ca74912b36fdd98237bb094766c606fe1618eee805a: Status 404 returned error can't find the container with id 8d6d7177b05683af7d811ca74912b36fdd98237bb094766c606fe1618eee805a
Oct 31 04:26:01.301621 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:01.301592    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"]
Oct 31 04:26:01.366530 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:01.366496    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb" event={"ID":"26655c2d-eaf1-482b-901f-8c52ec472f1d","Type":"ContainerStarted","Data":"8d6d7177b05683af7d811ca74912b36fdd98237bb094766c606fe1618eee805a"}
Oct 31 04:26:02.373452 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:02.373359    4900 generic.go:356] "Generic (PLEG): container finished" podID="26655c2d-eaf1-482b-901f-8c52ec472f1d" containerID="f6dd833bc62ee834dab576190885c7a2b9e4ce965fe3e71a13857b21b21e3059" exitCode=0
Oct 31 04:26:02.373873 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:02.373446    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb" event={"ID":"26655c2d-eaf1-482b-901f-8c52ec472f1d","Type":"ContainerDied","Data":"f6dd833bc62ee834dab576190885c7a2b9e4ce965fe3e71a13857b21b21e3059"}
Oct 31 04:26:03.065039 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:03.065004    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:03.284071 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:03.284031    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:03.564356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:03.564329    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"
Oct 31 04:26:03.587651 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:03.587617    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26655c2d-eaf1-482b-901f-8c52ec472f1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26655c2d-eaf1-482b-901f-8c52ec472f1d-maas-api-openshift-ca\") pod \"26655c2d-eaf1-482b-901f-8c52ec472f1d\" (UID: \"26655c2d-eaf1-482b-901f-8c52ec472f1d\") "
Oct 31 04:26:03.587850 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:03.587835    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26655c2d-eaf1-482b-901f-8c52ec472f1d-kube-api-access-5rntk\" (UniqueName: \"kubernetes.io/projected/26655c2d-eaf1-482b-901f-8c52ec472f1d-kube-api-access-5rntk\") pod \"26655c2d-eaf1-482b-901f-8c52ec472f1d\" (UID: \"26655c2d-eaf1-482b-901f-8c52ec472f1d\") "
Oct 31 04:26:03.587917 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:03.587878    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26655c2d-eaf1-482b-901f-8c52ec472f1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26655c2d-eaf1-482b-901f-8c52ec472f1d-maas-api-xks-ca\") pod \"26655c2d-eaf1-482b-901f-8c52ec472f1d\" (UID: \"26655c2d-eaf1-482b-901f-8c52ec472f1d\") "
Oct 31 04:26:03.587978 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:03.587957    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26655c2d-eaf1-482b-901f-8c52ec472f1d-maas-api-openshift-ca" pod "26655c2d-eaf1-482b-901f-8c52ec472f1d" (UID: "26655c2d-eaf1-482b-901f-8c52ec472f1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:26:03.588337 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:03.588313    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26655c2d-eaf1-482b-901f-8c52ec472f1d-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:03.589893 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:03.589865    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26655c2d-eaf1-482b-901f-8c52ec472f1d-maas-api-xks-ca" pod "26655c2d-eaf1-482b-901f-8c52ec472f1d" (UID: "26655c2d-eaf1-482b-901f-8c52ec472f1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:26:03.589975 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:03.589889    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26655c2d-eaf1-482b-901f-8c52ec472f1d-kube-api-access-5rntk" pod "26655c2d-eaf1-482b-901f-8c52ec472f1d" (UID: "26655c2d-eaf1-482b-901f-8c52ec472f1d"). InnerVolumeSpecName "kube-api-access-5rntk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:03.689113 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:03.689037    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rntk\" (UniqueName: \"kubernetes.io/projected/26655c2d-eaf1-482b-901f-8c52ec472f1d-kube-api-access-5rntk\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:03.689113 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:03.689066    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26655c2d-eaf1-482b-901f-8c52ec472f1d-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:04.386090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:04.386050    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"
Oct 31 04:26:04.386090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:04.386080    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb" event={"ID":"26655c2d-eaf1-482b-901f-8c52ec472f1d","Type":"ContainerDied","Data":"8d6d7177b05683af7d811ca74912b36fdd98237bb094766c606fe1618eee805a"}
Oct 31 04:26:04.386313 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:04.386127    4900 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d6d7177b05683af7d811ca74912b36fdd98237bb094766c606fe1618eee805a"
Oct 31 04:26:04.729273 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:04.729171    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:04.931655 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:04.931614    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:05.282315 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:05.282284    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"]
Oct 31 04:26:05.282496 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:05.282474    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2" podUID="fba5a14c-8608-4005-961d-869f98e5d3a7" containerName="istio-proxy" containerID="cri-o://3bc3cfa2d96cd57f1b29679ad196738b04f6c87c3d8ef6edf9df869fbfd8a875" gracePeriod=30
Oct 31 04:26:05.368156 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:05.368121    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:06.108211 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:06.108176    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:06.984063 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:26:06.984023    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:26:09.061243 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:09.061197    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:09.877020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:09.876979    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:10.426449 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.426415    4900 generic.go:356] "Generic (PLEG): container finished" podID="fba5a14c-8608-4005-961d-869f98e5d3a7" containerID="3bc3cfa2d96cd57f1b29679ad196738b04f6c87c3d8ef6edf9df869fbfd8a875" exitCode=0
Oct 31 04:26:10.426889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.426465    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2" event={"ID":"fba5a14c-8608-4005-961d-869f98e5d3a7","Type":"ContainerDied","Data":"3bc3cfa2d96cd57f1b29679ad196738b04f6c87c3d8ef6edf9df869fbfd8a875"}
Oct 31 04:26:10.426889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.426517    4900 scope.go:122] "RemoveContainer" containerID="a1994aebfe40728316c650405921475a95188999833b35fddc88feb3a6451a16"
Oct 31 04:26:10.539700 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.539674    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 04:26:10.577858 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.577759    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-data\") pod \"fba5a14c-8608-4005-961d-869f98e5d3a7\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") "
Oct 31 04:26:10.577858 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.577849    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba5a14c-8608-4005-961d-869f98e5d3a7-kube-api-access-wcq7h\" (UniqueName: \"kubernetes.io/projected/fba5a14c-8608-4005-961d-869f98e5d3a7-kube-api-access-wcq7h\") pod \"fba5a14c-8608-4005-961d-869f98e5d3a7\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") "
Oct 31 04:26:10.578078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.577902    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-credential-socket\") pod \"fba5a14c-8608-4005-961d-869f98e5d3a7\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") "
Oct 31 04:26:10.578078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.577933    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba5a14c-8608-4005-961d-869f98e5d3a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba5a14c-8608-4005-961d-869f98e5d3a7-istiod-ca-cert\") pod \"fba5a14c-8608-4005-961d-869f98e5d3a7\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") "
Oct 31 04:26:10.578078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.577965    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-envoy\") pod \"fba5a14c-8608-4005-961d-869f98e5d3a7\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") "
Oct 31 04:26:10.578078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.578017    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-workload-socket\") pod \"fba5a14c-8608-4005-961d-869f98e5d3a7\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") "
Oct 31 04:26:10.578078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.578026    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-data" pod "fba5a14c-8608-4005-961d-869f98e5d3a7" (UID: "fba5a14c-8608-4005-961d-869f98e5d3a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:26:10.578351 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.578165    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-podinfo\") pod \"fba5a14c-8608-4005-961d-869f98e5d3a7\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") "
Oct 31 04:26:10.578351 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.578255    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-token\" (UniqueName: \"kubernetes.io/projected/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-token\") pod \"fba5a14c-8608-4005-961d-869f98e5d3a7\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") "
Oct 31 04:26:10.578351 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.578264    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-credential-socket" pod "fba5a14c-8608-4005-961d-869f98e5d3a7" (UID: "fba5a14c-8608-4005-961d-869f98e5d3a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:26:10.578351 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.578286    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-workload-certs\") pod \"fba5a14c-8608-4005-961d-869f98e5d3a7\" (UID: \"fba5a14c-8608-4005-961d-869f98e5d3a7\") "
Oct 31 04:26:10.578351 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.578320    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-workload-socket" pod "fba5a14c-8608-4005-961d-869f98e5d3a7" (UID: "fba5a14c-8608-4005-961d-869f98e5d3a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:26:10.578591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.578393    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba5a14c-8608-4005-961d-869f98e5d3a7-istiod-ca-cert" pod "fba5a14c-8608-4005-961d-869f98e5d3a7" (UID: "fba5a14c-8608-4005-961d-869f98e5d3a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:26:10.578648 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.578603    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-workload-certs" pod "fba5a14c-8608-4005-961d-869f98e5d3a7" (UID: "fba5a14c-8608-4005-961d-869f98e5d3a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:26:10.578885 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.578864    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:10.578947 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.578892    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:10.578947 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.578909    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fba5a14c-8608-4005-961d-869f98e5d3a7-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:10.578947 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.578923    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:10.578947 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.578937    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:10.580287 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.580262    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-token" pod "fba5a14c-8608-4005-961d-869f98e5d3a7" (UID: "fba5a14c-8608-4005-961d-869f98e5d3a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:10.580402 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.580306    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-envoy" pod "fba5a14c-8608-4005-961d-869f98e5d3a7" (UID: "fba5a14c-8608-4005-961d-869f98e5d3a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:26:10.580590 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.580571    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba5a14c-8608-4005-961d-869f98e5d3a7-kube-api-access-wcq7h" pod "fba5a14c-8608-4005-961d-869f98e5d3a7" (UID: "fba5a14c-8608-4005-961d-869f98e5d3a7"). InnerVolumeSpecName "kube-api-access-wcq7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:10.580673 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.580610    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-podinfo" pod "fba5a14c-8608-4005-961d-869f98e5d3a7" (UID: "fba5a14c-8608-4005-961d-869f98e5d3a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:26:10.679764 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.679733    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:10.679764 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.679758    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:10.679764 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.679768    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fba5a14c-8608-4005-961d-869f98e5d3a7-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:10.680003 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:10.679777    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcq7h\" (UniqueName: \"kubernetes.io/projected/fba5a14c-8608-4005-961d-869f98e5d3a7-kube-api-access-wcq7h\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:11.290055 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.290019    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"]
Oct 31 04:26:11.290578 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.290564    4900 cpu_manager.go:414] "containerMap: removing container" podUID="26655c2d-eaf1-482b-901f-8c52ec472f1d" containerName="revoke-keys"
Oct 31 04:26:11.290625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.290581    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26655c2d-eaf1-482b-901f-8c52ec472f1d" containerName="revoke-keys"
Oct 31 04:26:11.290625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.290600    4900 cpu_manager.go:414] "containerMap: removing container" podUID="fba5a14c-8608-4005-961d-869f98e5d3a7" containerName="istio-proxy"
Oct 31 04:26:11.290625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.290605    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba5a14c-8608-4005-961d-869f98e5d3a7" containerName="istio-proxy"
Oct 31 04:26:11.290730 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.290706    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="26655c2d-eaf1-482b-901f-8c52ec472f1d" containerName="revoke-keys"
Oct 31 04:26:11.290730 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.290716    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba5a14c-8608-4005-961d-869f98e5d3a7" containerName="istio-proxy"
Oct 31 04:26:11.290730 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.290726    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba5a14c-8608-4005-961d-869f98e5d3a7" containerName="istio-proxy"
Oct 31 04:26:11.294309 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.294292    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"
Oct 31 04:26:11.362505 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.362468    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"]
Oct 31 04:26:11.387061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.387023    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-892zv\" (UniqueName: \"kubernetes.io/projected/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-kube-api-access-892zv\") pod \"maas-api-revoke-sub-d57466af-srsjc\" (UID: \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"
Oct 31 04:26:11.387223 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.387150    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d57466af-srsjc\" (UID: \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"
Oct 31 04:26:11.387322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.387299    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d57466af-srsjc\" (UID: \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"
Oct 31 04:26:11.409167 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.409140    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:26:11.409167 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.409168    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:26:11.433986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.433907    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2" event={"ID":"fba5a14c-8608-4005-961d-869f98e5d3a7","Type":"ContainerDied","Data":"14b0d754c998f85c44289e46ba053c6994b2f2749e67c77bf1d4377a4d42d1be"}
Oct 31 04:26:11.433986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.433931    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"
Oct 31 04:26:11.433986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.433967    4900 scope.go:122] "RemoveContainer" containerID="3bc3cfa2d96cd57f1b29679ad196738b04f6c87c3d8ef6edf9df869fbfd8a875"
Oct 31 04:26:11.488550 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.488522    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-892zv\" (UniqueName: \"kubernetes.io/projected/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-kube-api-access-892zv\") pod \"maas-api-revoke-sub-d57466af-srsjc\" (UID: \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"
Oct 31 04:26:11.488692 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.488564    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d57466af-srsjc\" (UID: \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"
Oct 31 04:26:11.488692 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.488628    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d57466af-srsjc\" (UID: \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"
Oct 31 04:26:11.489202 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.489183    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d57466af-srsjc\" (UID: \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"
Oct 31 04:26:11.490941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.490915    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d57466af-srsjc\" (UID: \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"
Oct 31 04:26:11.545072 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.545042    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-892zv\" (UniqueName: \"kubernetes.io/projected/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-kube-api-access-892zv\") pod \"maas-api-revoke-sub-d57466af-srsjc\" (UID: \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"
Oct 31 04:26:11.598416 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.598380    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"]
Oct 31 04:26:11.602423 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.602393    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-861dfd-openshift-default-7674fcd955-jwqp2"]
Oct 31 04:26:11.607393 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.607374    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"
Oct 31 04:26:11.607528 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.607511    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"]
Oct 31 04:26:11.975669 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:26:11.975624    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e1d05c_d6da_4ca6_bc56_ea8e9e893f9b.slice/crio-12d7e8f19f062e925287d926599fa8cc6c8e4bea75bd91575b505a5a9aa85b5b WatchSource:0}: Error finding container 12d7e8f19f062e925287d926599fa8cc6c8e4bea75bd91575b505a5a9aa85b5b: Status 404 returned error can't find the container with id 12d7e8f19f062e925287d926599fa8cc6c8e4bea75bd91575b505a5a9aa85b5b
Oct 31 04:26:11.978155 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:11.978128    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"]
Oct 31 04:26:12.039208 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:12.039182    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:12.304985 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:12.304948    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:12.443229 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:12.443191    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc" event={"ID":"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b","Type":"ContainerStarted","Data":"dd9758c1b28209903758ae4f564eb9959fbacdb3631576199629e2d5b145cf32"}
Oct 31 04:26:12.443668 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:12.443247    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc" event={"ID":"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b","Type":"ContainerStarted","Data":"12d7e8f19f062e925287d926599fa8cc6c8e4bea75bd91575b505a5a9aa85b5b"}
Oct 31 04:26:12.559475 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:12.559377    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc" podStartSLOduration=1.559363203 podStartE2EDuration="1.559363203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:26:12.556521217 +0000 UTC m=+4725.522854561" watchObservedRunningTime="2026-10-31 04:26:12.559363203 +0000 UTC m=+4725.525696546"
Oct 31 04:26:12.811959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:12.811893    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:13.152814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:13.152713    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba5a14c-8608-4005-961d-869f98e5d3a7" path="/var/lib/kubelet/pods/fba5a14c-8608-4005-961d-869f98e5d3a7/volumes"
Oct 31 04:26:13.450698 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:13.450620    4900 generic.go:356] "Generic (PLEG): container finished" podID="33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b" containerID="dd9758c1b28209903758ae4f564eb9959fbacdb3631576199629e2d5b145cf32" exitCode=0
Oct 31 04:26:13.450698 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:13.450663    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc" event={"ID":"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b","Type":"ContainerDied","Data":"dd9758c1b28209903758ae4f564eb9959fbacdb3631576199629e2d5b145cf32"}
Oct 31 04:26:14.596500 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:14.596475    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"
Oct 31 04:26:14.724691 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:14.724583    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-kube-api-access-892zv\" (UniqueName: \"kubernetes.io/projected/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-kube-api-access-892zv\") pod \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\" (UID: \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\") "
Oct 31 04:26:14.724889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:14.724728    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-maas-api-xks-ca\") pod \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\" (UID: \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\") "
Oct 31 04:26:14.724889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:14.724829    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-maas-api-openshift-ca\") pod \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\" (UID: \"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b\") "
Oct 31 04:26:14.725192 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:14.725170    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-maas-api-openshift-ca" pod "33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b" (UID: "33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:26:14.725424 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:14.725406    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:14.727239 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:14.727215    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-maas-api-xks-ca" pod "33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b" (UID: "33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:26:14.727353 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:14.727245    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-kube-api-access-892zv" pod "33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b" (UID: "33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b"). InnerVolumeSpecName "kube-api-access-892zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:14.826211 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:14.826180    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-892zv\" (UniqueName: \"kubernetes.io/projected/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-kube-api-access-892zv\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:14.826211 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:14.826209    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:14.984346 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:14.984318    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:15.463433 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:15.463357    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"
Oct 31 04:26:15.463433 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:15.463386    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc" event={"ID":"33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b","Type":"ContainerDied","Data":"12d7e8f19f062e925287d926599fa8cc6c8e4bea75bd91575b505a5a9aa85b5b"}
Oct 31 04:26:15.463622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:15.463446    4900 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12d7e8f19f062e925287d926599fa8cc6c8e4bea75bd91575b505a5a9aa85b5b"
Oct 31 04:26:16.608218 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:16.608177    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62" podUID="4c9a50ed-4995-42b7-b5f9-76dead1711c7" containerName="payload-processing" containerID="cri-o://bb035c7bfb44ce61a937def286ab378764cd5cde775514005104ac829ae19640" gracePeriod=25
Oct 31 04:26:16.734989 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:16.734956    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:18.472468 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:18.472434    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:19.913975 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:19.913939    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:20.821172 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:20.821129    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62" podUID="4c9a50ed-4995-42b7-b5f9-76dead1711c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:26:23.177161 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:23.177130    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:23.394431 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:23.394398    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"]
Oct 31 04:26:23.487047 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:23.487012    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:25.008136 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:25.008106    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:25.459642 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:25.459553    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"]
Oct 31 04:26:25.601874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:25.601839    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"]
Oct 31 04:26:25.925075 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:25.925036    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:26.123989 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:26.123954    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:28.703897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:28.703864    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:29.906839 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:29.906807    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:30.460239 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:30.460193    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz" podUID="0e83740c-ab5a-49f8-9d97-1fadd09b65e4" containerName="payload-processing" containerID="cri-o://8c187be809aded781cad1814615bed525b7b76c38b50ca1c1603abfce6a00a3c" gracePeriod=25
Oct 31 04:26:30.602398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:30.602361    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr" podUID="8722d22a-ef10-4b87-9616-80299e5a6279" containerName="payload-pre-processing" containerID="cri-o://46bc0609f6c92ac1a2b176306a3789cf43f514c535b429e8d75b5e3cc0892a33" gracePeriod=25
Oct 31 04:26:30.731509 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:30.731482    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"
Oct 31 04:26:30.821227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:30.821189    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62" podUID="4c9a50ed-4995-42b7-b5f9-76dead1711c7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:26:30.847549 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:30.847519    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"
Oct 31 04:26:30.916185 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:30.916153    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e83740c-ab5a-49f8-9d97-1fadd09b65e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e83740c-ab5a-49f8-9d97-1fadd09b65e4-plugins-config-volume\") pod \"0e83740c-ab5a-49f8-9d97-1fadd09b65e4\" (UID: \"0e83740c-ab5a-49f8-9d97-1fadd09b65e4\") "
Oct 31 04:26:30.916588 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:30.916290    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e83740c-ab5a-49f8-9d97-1fadd09b65e4-kube-api-access-5cd9r\" (UniqueName: \"kubernetes.io/projected/0e83740c-ab5a-49f8-9d97-1fadd09b65e4-kube-api-access-5cd9r\") pod \"0e83740c-ab5a-49f8-9d97-1fadd09b65e4\" (UID: \"0e83740c-ab5a-49f8-9d97-1fadd09b65e4\") "
Oct 31 04:26:30.916644 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:30.916586    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e83740c-ab5a-49f8-9d97-1fadd09b65e4-plugins-config-volume" pod "0e83740c-ab5a-49f8-9d97-1fadd09b65e4" (UID: "0e83740c-ab5a-49f8-9d97-1fadd09b65e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:26:30.916693 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:30.916682    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e83740c-ab5a-49f8-9d97-1fadd09b65e4-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:30.918330 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:30.918311    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e83740c-ab5a-49f8-9d97-1fadd09b65e4-kube-api-access-5cd9r" pod "0e83740c-ab5a-49f8-9d97-1fadd09b65e4" (UID: "0e83740c-ab5a-49f8-9d97-1fadd09b65e4"). InnerVolumeSpecName "kube-api-access-5cd9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:31.017198 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.017166    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8722d22a-ef10-4b87-9616-80299e5a6279-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8722d22a-ef10-4b87-9616-80299e5a6279-plugins-config-volume\") pod \"8722d22a-ef10-4b87-9616-80299e5a6279\" (UID: \"8722d22a-ef10-4b87-9616-80299e5a6279\") "
Oct 31 04:26:31.017416 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.017254    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8722d22a-ef10-4b87-9616-80299e5a6279-kube-api-access-jtfv2\" (UniqueName: \"kubernetes.io/projected/8722d22a-ef10-4b87-9616-80299e5a6279-kube-api-access-jtfv2\") pod \"8722d22a-ef10-4b87-9616-80299e5a6279\" (UID: \"8722d22a-ef10-4b87-9616-80299e5a6279\") "
Oct 31 04:26:31.017562 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.017537    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8722d22a-ef10-4b87-9616-80299e5a6279-plugins-config-volume" pod "8722d22a-ef10-4b87-9616-80299e5a6279" (UID: "8722d22a-ef10-4b87-9616-80299e5a6279"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:26:31.017562 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.017561    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cd9r\" (UniqueName: \"kubernetes.io/projected/0e83740c-ab5a-49f8-9d97-1fadd09b65e4-kube-api-access-5cd9r\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:31.019471 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.019443    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8722d22a-ef10-4b87-9616-80299e5a6279-kube-api-access-jtfv2" pod "8722d22a-ef10-4b87-9616-80299e5a6279" (UID: "8722d22a-ef10-4b87-9616-80299e5a6279"). InnerVolumeSpecName "kube-api-access-jtfv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:31.118579 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.118545    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8722d22a-ef10-4b87-9616-80299e5a6279-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:31.118579 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.118574    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtfv2\" (UniqueName: \"kubernetes.io/projected/8722d22a-ef10-4b87-9616-80299e5a6279-kube-api-access-jtfv2\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:31.563409 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.563368    4900 generic.go:356] "Generic (PLEG): container finished" podID="8722d22a-ef10-4b87-9616-80299e5a6279" containerID="46bc0609f6c92ac1a2b176306a3789cf43f514c535b429e8d75b5e3cc0892a33" exitCode=0
Oct 31 04:26:31.563582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.563416    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"
Oct 31 04:26:31.563582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.563461    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr" event={"ID":"8722d22a-ef10-4b87-9616-80299e5a6279","Type":"ContainerDied","Data":"46bc0609f6c92ac1a2b176306a3789cf43f514c535b429e8d75b5e3cc0892a33"}
Oct 31 04:26:31.563582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.563520    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr" event={"ID":"8722d22a-ef10-4b87-9616-80299e5a6279","Type":"ContainerDied","Data":"0ce1f0c82227e0c4ede0d0c35cea82e0610ac23cdb31cf6a917e2933f8ac3294"}
Oct 31 04:26:31.563582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.563551    4900 scope.go:122] "RemoveContainer" containerID="46bc0609f6c92ac1a2b176306a3789cf43f514c535b429e8d75b5e3cc0892a33"
Oct 31 04:26:31.566531 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.565741    4900 generic.go:356] "Generic (PLEG): container finished" podID="0e83740c-ab5a-49f8-9d97-1fadd09b65e4" containerID="8c187be809aded781cad1814615bed525b7b76c38b50ca1c1603abfce6a00a3c" exitCode=0
Oct 31 04:26:31.566531 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.565857    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz" event={"ID":"0e83740c-ab5a-49f8-9d97-1fadd09b65e4","Type":"ContainerDied","Data":"8c187be809aded781cad1814615bed525b7b76c38b50ca1c1603abfce6a00a3c"}
Oct 31 04:26:31.566531 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.565897    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz" event={"ID":"0e83740c-ab5a-49f8-9d97-1fadd09b65e4","Type":"ContainerDied","Data":"2cc573ba4e81bda2ecf2363384ab312a6a3115f1dd9175e426c3e1a127b617a6"}
Oct 31 04:26:31.566531 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.565963    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"
Oct 31 04:26:31.574480 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.574462    4900 scope.go:122] "RemoveContainer" containerID="46bc0609f6c92ac1a2b176306a3789cf43f514c535b429e8d75b5e3cc0892a33"
Oct 31 04:26:31.574747 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:26:31.574726    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46bc0609f6c92ac1a2b176306a3789cf43f514c535b429e8d75b5e3cc0892a33\": container with ID starting with 46bc0609f6c92ac1a2b176306a3789cf43f514c535b429e8d75b5e3cc0892a33 not found: ID does not exist" containerID="46bc0609f6c92ac1a2b176306a3789cf43f514c535b429e8d75b5e3cc0892a33"
Oct 31 04:26:31.574816 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.574756    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bc0609f6c92ac1a2b176306a3789cf43f514c535b429e8d75b5e3cc0892a33"} err="failed to get container status \"46bc0609f6c92ac1a2b176306a3789cf43f514c535b429e8d75b5e3cc0892a33\": rpc error: code = NotFound desc = could not find container \"46bc0609f6c92ac1a2b176306a3789cf43f514c535b429e8d75b5e3cc0892a33\": container with ID starting with 46bc0609f6c92ac1a2b176306a3789cf43f514c535b429e8d75b5e3cc0892a33 not found: ID does not exist"
Oct 31 04:26:31.574816 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.574774    4900 scope.go:122] "RemoveContainer" containerID="8c187be809aded781cad1814615bed525b7b76c38b50ca1c1603abfce6a00a3c"
Oct 31 04:26:31.582831 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.582815    4900 scope.go:122] "RemoveContainer" containerID="8c187be809aded781cad1814615bed525b7b76c38b50ca1c1603abfce6a00a3c"
Oct 31 04:26:31.583070 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:26:31.583051    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c187be809aded781cad1814615bed525b7b76c38b50ca1c1603abfce6a00a3c\": container with ID starting with 8c187be809aded781cad1814615bed525b7b76c38b50ca1c1603abfce6a00a3c not found: ID does not exist" containerID="8c187be809aded781cad1814615bed525b7b76c38b50ca1c1603abfce6a00a3c"
Oct 31 04:26:31.583116 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.583078    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c187be809aded781cad1814615bed525b7b76c38b50ca1c1603abfce6a00a3c"} err="failed to get container status \"8c187be809aded781cad1814615bed525b7b76c38b50ca1c1603abfce6a00a3c\": rpc error: code = NotFound desc = could not find container \"8c187be809aded781cad1814615bed525b7b76c38b50ca1c1603abfce6a00a3c\": container with ID starting with 8c187be809aded781cad1814615bed525b7b76c38b50ca1c1603abfce6a00a3c not found: ID does not exist"
Oct 31 04:26:31.727940 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.727856    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"]
Oct 31 04:26:31.777909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.777876    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-26c74d32-b558f8d67-kvxgr"]
Oct 31 04:26:31.841777 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.841742    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"]
Oct 31 04:26:31.844634 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:31.844607    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-26c74d32-6747c4bc7c-llhgz"]
Oct 31 04:26:33.152568 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:33.152537    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e83740c-ab5a-49f8-9d97-1fadd09b65e4" path="/var/lib/kubelet/pods/0e83740c-ab5a-49f8-9d97-1fadd09b65e4/volumes"
Oct 31 04:26:33.152986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:33.152977    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8722d22a-ef10-4b87-9616-80299e5a6279" path="/var/lib/kubelet/pods/8722d22a-ef10-4b87-9616-80299e5a6279/volumes"
Oct 31 04:26:34.665538 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:34.665499    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:36.872331 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:36.872278    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"
Oct 31 04:26:36.983317 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:36.983282    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9a50ed-4995-42b7-b5f9-76dead1711c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9a50ed-4995-42b7-b5f9-76dead1711c7-plugins-config-volume\") pod \"4c9a50ed-4995-42b7-b5f9-76dead1711c7\" (UID: \"4c9a50ed-4995-42b7-b5f9-76dead1711c7\") "
Oct 31 04:26:36.983514 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:36.983445    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9a50ed-4995-42b7-b5f9-76dead1711c7-kube-api-access-2xsqz\" (UniqueName: \"kubernetes.io/projected/4c9a50ed-4995-42b7-b5f9-76dead1711c7-kube-api-access-2xsqz\") pod \"4c9a50ed-4995-42b7-b5f9-76dead1711c7\" (UID: \"4c9a50ed-4995-42b7-b5f9-76dead1711c7\") "
Oct 31 04:26:36.983703 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:36.983678    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9a50ed-4995-42b7-b5f9-76dead1711c7-plugins-config-volume" pod "4c9a50ed-4995-42b7-b5f9-76dead1711c7" (UID: "4c9a50ed-4995-42b7-b5f9-76dead1711c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:26:36.984186 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:36.984014    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9a50ed-4995-42b7-b5f9-76dead1711c7-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:36.986348 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:36.986320    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9a50ed-4995-42b7-b5f9-76dead1711c7-kube-api-access-2xsqz" pod "4c9a50ed-4995-42b7-b5f9-76dead1711c7" (UID: "4c9a50ed-4995-42b7-b5f9-76dead1711c7"). InnerVolumeSpecName "kube-api-access-2xsqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:36.990780 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:26:36.990746    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:26:37.085227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:37.085191    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xsqz\" (UniqueName: \"kubernetes.io/projected/4c9a50ed-4995-42b7-b5f9-76dead1711c7-kube-api-access-2xsqz\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:37.601424 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:37.601387    4900 generic.go:356] "Generic (PLEG): container finished" podID="4c9a50ed-4995-42b7-b5f9-76dead1711c7" containerID="bb035c7bfb44ce61a937def286ab378764cd5cde775514005104ac829ae19640" exitCode=0
Oct 31 04:26:37.601611 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:37.601432    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"
Oct 31 04:26:37.601611 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:37.601457    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62" event={"ID":"4c9a50ed-4995-42b7-b5f9-76dead1711c7","Type":"ContainerDied","Data":"bb035c7bfb44ce61a937def286ab378764cd5cde775514005104ac829ae19640"}
Oct 31 04:26:37.601611 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:37.601504    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62" event={"ID":"4c9a50ed-4995-42b7-b5f9-76dead1711c7","Type":"ContainerDied","Data":"0bab9d654f6c9a584fe187c6a9ddbdfb851bb434251ce8d892809f1c6fe80a06"}
Oct 31 04:26:37.601611 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:37.601524    4900 scope.go:122] "RemoveContainer" containerID="bb035c7bfb44ce61a937def286ab378764cd5cde775514005104ac829ae19640"
Oct 31 04:26:37.610845 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:37.610825    4900 scope.go:122] "RemoveContainer" containerID="bb035c7bfb44ce61a937def286ab378764cd5cde775514005104ac829ae19640"
Oct 31 04:26:37.611107 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:26:37.611088    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb035c7bfb44ce61a937def286ab378764cd5cde775514005104ac829ae19640\": container with ID starting with bb035c7bfb44ce61a937def286ab378764cd5cde775514005104ac829ae19640 not found: ID does not exist" containerID="bb035c7bfb44ce61a937def286ab378764cd5cde775514005104ac829ae19640"
Oct 31 04:26:37.611154 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:37.611115    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb035c7bfb44ce61a937def286ab378764cd5cde775514005104ac829ae19640"} err="failed to get container status \"bb035c7bfb44ce61a937def286ab378764cd5cde775514005104ac829ae19640\": rpc error: code = NotFound desc = could not find container \"bb035c7bfb44ce61a937def286ab378764cd5cde775514005104ac829ae19640\": container with ID starting with bb035c7bfb44ce61a937def286ab378764cd5cde775514005104ac829ae19640 not found: ID does not exist"
Oct 31 04:26:37.765480 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:37.765447    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"]
Oct 31 04:26:37.815246 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:37.815207    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a5f70c-5f89d5474f-p2x62"]
Oct 31 04:26:38.414106 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.414046    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98" podUID="ade0cb97-d48a-4a37-9a3e-96ea531fbf46" containerName="main" containerID="cri-o://90428b3af54d6d3fee71e3933f400386e2fa92d563c9873a34d24a784a427d03" gracePeriod=45
Oct 31 04:26:38.611635 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.611605    4900 generic.go:356] "Generic (PLEG): container finished" podID="ade0cb97-d48a-4a37-9a3e-96ea531fbf46" containerID="90428b3af54d6d3fee71e3933f400386e2fa92d563c9873a34d24a784a427d03" exitCode=0
Oct 31 04:26:38.611859 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.611669    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98" event={"ID":"ade0cb97-d48a-4a37-9a3e-96ea531fbf46","Type":"ContainerDied","Data":"90428b3af54d6d3fee71e3933f400386e2fa92d563c9873a34d24a784a427d03"}
Oct 31 04:26:38.675291 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.675221    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:26:38.805053 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.805014    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-model-cache\") pod \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") "
Oct 31 04:26:38.805231 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.805089    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-tmp-dir\") pod \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") "
Oct 31 04:26:38.805231 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.805127    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-kube-api-access-5lmck\" (UniqueName: \"kubernetes.io/projected/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-kube-api-access-5lmck\") pod \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") "
Oct 31 04:26:38.805374 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.805255    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-home\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-home\") pod \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") "
Oct 31 04:26:38.805374 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.805321    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-dshm\") pod \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") "
Oct 31 04:26:38.805374 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.805348    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-tmp-dir" pod "ade0cb97-d48a-4a37-9a3e-96ea531fbf46" (UID: "ade0cb97-d48a-4a37-9a3e-96ea531fbf46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:26:38.805374 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.805366    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-model-cache" pod "ade0cb97-d48a-4a37-9a3e-96ea531fbf46" (UID: "ade0cb97-d48a-4a37-9a3e-96ea531fbf46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:26:38.805602 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.805357    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-tls-certs\" (UniqueName: \"kubernetes.io/secret/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-tls-certs\") pod \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\" (UID: \"ade0cb97-d48a-4a37-9a3e-96ea531fbf46\") "
Oct 31 04:26:38.805712 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.805685    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-home" pod "ade0cb97-d48a-4a37-9a3e-96ea531fbf46" (UID: "ade0cb97-d48a-4a37-9a3e-96ea531fbf46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:26:38.806066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.806001    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:38.806066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.806028    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:38.806066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.806041    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:38.807526 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.807502    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-kube-api-access-5lmck" pod "ade0cb97-d48a-4a37-9a3e-96ea531fbf46" (UID: "ade0cb97-d48a-4a37-9a3e-96ea531fbf46"). InnerVolumeSpecName "kube-api-access-5lmck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:26:38.807915 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.807894    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-tls-certs" pod "ade0cb97-d48a-4a37-9a3e-96ea531fbf46" (UID: "ade0cb97-d48a-4a37-9a3e-96ea531fbf46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:26:38.808002 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.807899    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-dshm" pod "ade0cb97-d48a-4a37-9a3e-96ea531fbf46" (UID: "ade0cb97-d48a-4a37-9a3e-96ea531fbf46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:26:38.907174 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.907131    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lmck\" (UniqueName: \"kubernetes.io/projected/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-kube-api-access-5lmck\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:38.907174 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.907166    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:38.907174 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:38.907176    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ade0cb97-d48a-4a37-9a3e-96ea531fbf46-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:26:39.152978 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:39.152945    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9a50ed-4995-42b7-b5f9-76dead1711c7" path="/var/lib/kubelet/pods/4c9a50ed-4995-42b7-b5f9-76dead1711c7/volumes"
Oct 31 04:26:39.451985 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:39.451901    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:39.618697 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:39.618664    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98" event={"ID":"ade0cb97-d48a-4a37-9a3e-96ea531fbf46","Type":"ContainerDied","Data":"036ff4ff382a4adef6c46022ff38128093db2d763af8447fe4a2debe1a47c7a2"}
Oct 31 04:26:39.618697 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:39.618691    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"
Oct 31 04:26:39.618697 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:39.618714    4900 scope.go:122] "RemoveContainer" containerID="90428b3af54d6d3fee71e3933f400386e2fa92d563c9873a34d24a784a427d03"
Oct 31 04:26:39.784341 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:39.784306    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"]
Oct 31 04:26:39.832910 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:39.832877    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2339ed/rate-test-model-2339ed-kserve-9974f87b5-7vc98"]
Oct 31 04:26:41.152230 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:41.152198    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade0cb97-d48a-4a37-9a3e-96ea531fbf46" path="/var/lib/kubelet/pods/ade0cb97-d48a-4a37-9a3e-96ea531fbf46/volumes"
Oct 31 04:26:44.199689 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:44.199652    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:48.961208 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:48.961144    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:53.090041 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:53.090000    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:54.431781 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:54.431746    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:54.861015 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:54.860979    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:58.895075 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:58.895035    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:59.118441 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:59.118406    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:59.635870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:59.635835    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"]
Oct 31 04:26:59.761269 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:59.761230    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:26:59.969425 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:26:59.969336    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:00.458261 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:00.458231    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:00.942363 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:00.942323    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:01.168883 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.168850    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:01.409541 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.409506    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:01.594634 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.594597    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"]
Oct 31 04:27:01.666979 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.666880    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:01.928879 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.928739    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:01.951912 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.951878    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"]
Oct 31 04:27:01.952408 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952394    4900 cpu_manager.go:414] "containerMap: removing container" podUID="4c9a50ed-4995-42b7-b5f9-76dead1711c7" containerName="payload-processing"
Oct 31 04:27:01.952478 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952410    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9a50ed-4995-42b7-b5f9-76dead1711c7" containerName="payload-processing"
Oct 31 04:27:01.952478 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952426    4900 cpu_manager.go:414] "containerMap: removing container" podUID="fba5a14c-8608-4005-961d-869f98e5d3a7" containerName="istio-proxy"
Oct 31 04:27:01.952478 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952432    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba5a14c-8608-4005-961d-869f98e5d3a7" containerName="istio-proxy"
Oct 31 04:27:01.952478 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952444    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0e83740c-ab5a-49f8-9d97-1fadd09b65e4" containerName="payload-processing"
Oct 31 04:27:01.952478 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952452    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e83740c-ab5a-49f8-9d97-1fadd09b65e4" containerName="payload-processing"
Oct 31 04:27:01.952478 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952473    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ade0cb97-d48a-4a37-9a3e-96ea531fbf46" containerName="main"
Oct 31 04:27:01.952478 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952478    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade0cb97-d48a-4a37-9a3e-96ea531fbf46" containerName="main"
Oct 31 04:27:01.952736 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952486    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8722d22a-ef10-4b87-9616-80299e5a6279" containerName="payload-pre-processing"
Oct 31 04:27:01.952736 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952490    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8722d22a-ef10-4b87-9616-80299e5a6279" containerName="payload-pre-processing"
Oct 31 04:27:01.952736 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952504    4900 cpu_manager.go:414] "containerMap: removing container" podUID="33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b" containerName="revoke-keys"
Oct 31 04:27:01.952736 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952510    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b" containerName="revoke-keys"
Oct 31 04:27:01.952736 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952586    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9a50ed-4995-42b7-b5f9-76dead1711c7" containerName="payload-processing"
Oct 31 04:27:01.952736 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952599    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e83740c-ab5a-49f8-9d97-1fadd09b65e4" containerName="payload-processing"
Oct 31 04:27:01.952736 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952609    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade0cb97-d48a-4a37-9a3e-96ea531fbf46" containerName="main"
Oct 31 04:27:01.952736 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952615    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8722d22a-ef10-4b87-9616-80299e5a6279" containerName="payload-pre-processing"
Oct 31 04:27:01.952736 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.952623    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b" containerName="revoke-keys"
Oct 31 04:27:01.956216 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.956195    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"
Oct 31 04:27:01.981914 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.981876    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxmlx\" (UniqueName: \"kubernetes.io/projected/f38ad177-3298-48f2-9ec0-18a4a438a664-kube-api-access-wxmlx\") pod \"maas-api-revoke-sub-dbced6ba-bm8tv\" (UID: \"f38ad177-3298-48f2-9ec0-18a4a438a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"
Oct 31 04:27:01.981914 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.981916    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f38ad177-3298-48f2-9ec0-18a4a438a664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbced6ba-bm8tv\" (UID: \"f38ad177-3298-48f2-9ec0-18a4a438a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"
Oct 31 04:27:01.982125 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:01.982104    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f38ad177-3298-48f2-9ec0-18a4a438a664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbced6ba-bm8tv\" (UID: \"f38ad177-3298-48f2-9ec0-18a4a438a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"
Oct 31 04:27:02.008264 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:02.008228    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:27:02.008532 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:02.008439    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b727f\""
Oct 31 04:27:02.009374 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:02.009343    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"]
Oct 31 04:27:02.083180 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:02.083143    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f38ad177-3298-48f2-9ec0-18a4a438a664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbced6ba-bm8tv\" (UID: \"f38ad177-3298-48f2-9ec0-18a4a438a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"
Oct 31 04:27:02.083393 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:02.083244    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxmlx\" (UniqueName: \"kubernetes.io/projected/f38ad177-3298-48f2-9ec0-18a4a438a664-kube-api-access-wxmlx\") pod \"maas-api-revoke-sub-dbced6ba-bm8tv\" (UID: \"f38ad177-3298-48f2-9ec0-18a4a438a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"
Oct 31 04:27:02.083393 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:02.083277    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f38ad177-3298-48f2-9ec0-18a4a438a664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbced6ba-bm8tv\" (UID: \"f38ad177-3298-48f2-9ec0-18a4a438a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"
Oct 31 04:27:02.083881 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:02.083854    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f38ad177-3298-48f2-9ec0-18a4a438a664-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbced6ba-bm8tv\" (UID: \"f38ad177-3298-48f2-9ec0-18a4a438a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"
Oct 31 04:27:02.085755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:02.085727    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f38ad177-3298-48f2-9ec0-18a4a438a664-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbced6ba-bm8tv\" (UID: \"f38ad177-3298-48f2-9ec0-18a4a438a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"
Oct 31 04:27:02.133011 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:02.132969    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:02.139753 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:02.139718    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxmlx\" (UniqueName: \"kubernetes.io/projected/f38ad177-3298-48f2-9ec0-18a4a438a664-kube-api-access-wxmlx\") pod \"maas-api-revoke-sub-dbced6ba-bm8tv\" (UID: \"f38ad177-3298-48f2-9ec0-18a4a438a664\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"
Oct 31 04:27:02.270846 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:02.270812    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"
Oct 31 04:27:02.639357 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:27:02.639294    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf38ad177_3298_48f2_9ec0_18a4a438a664.slice/crio-acc195aa493434d75f72dd67186db65e96905211efc0f11940d7ed0f0457658b WatchSource:0}: Error finding container acc195aa493434d75f72dd67186db65e96905211efc0f11940d7ed0f0457658b: Status 404 returned error can't find the container with id acc195aa493434d75f72dd67186db65e96905211efc0f11940d7ed0f0457658b
Oct 31 04:27:02.641781 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:02.641753    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"]
Oct 31 04:27:02.750811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:02.750746    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv" event={"ID":"f38ad177-3298-48f2-9ec0-18a4a438a664","Type":"ContainerStarted","Data":"acc195aa493434d75f72dd67186db65e96905211efc0f11940d7ed0f0457658b"}
Oct 31 04:27:03.758494 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:03.758460    4900 generic.go:356] "Generic (PLEG): container finished" podID="f38ad177-3298-48f2-9ec0-18a4a438a664" containerID="f865a90fb21b88197b8bf82610ad79372167aa38d85ed64a33177b7505c41435" exitCode=0
Oct 31 04:27:03.758951 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:03.758546    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv" event={"ID":"f38ad177-3298-48f2-9ec0-18a4a438a664","Type":"ContainerDied","Data":"f865a90fb21b88197b8bf82610ad79372167aa38d85ed64a33177b7505c41435"}
Oct 31 04:27:04.242239 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:04.242197    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:04.894450 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:04.894419    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"
Oct 31 04:27:05.021411 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:05.021324    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f38ad177-3298-48f2-9ec0-18a4a438a664-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f38ad177-3298-48f2-9ec0-18a4a438a664-maas-api-openshift-ca\") pod \"f38ad177-3298-48f2-9ec0-18a4a438a664\" (UID: \"f38ad177-3298-48f2-9ec0-18a4a438a664\") "
Oct 31 04:27:05.021592 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:05.021456    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f38ad177-3298-48f2-9ec0-18a4a438a664-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f38ad177-3298-48f2-9ec0-18a4a438a664-maas-api-xks-ca\") pod \"f38ad177-3298-48f2-9ec0-18a4a438a664\" (UID: \"f38ad177-3298-48f2-9ec0-18a4a438a664\") "
Oct 31 04:27:05.021663 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:05.021605    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38ad177-3298-48f2-9ec0-18a4a438a664-kube-api-access-wxmlx\" (UniqueName: \"kubernetes.io/projected/f38ad177-3298-48f2-9ec0-18a4a438a664-kube-api-access-wxmlx\") pod \"f38ad177-3298-48f2-9ec0-18a4a438a664\" (UID: \"f38ad177-3298-48f2-9ec0-18a4a438a664\") "
Oct 31 04:27:05.021761 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:05.021729    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38ad177-3298-48f2-9ec0-18a4a438a664-maas-api-openshift-ca" pod "f38ad177-3298-48f2-9ec0-18a4a438a664" (UID: "f38ad177-3298-48f2-9ec0-18a4a438a664"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:05.022037 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:05.022018    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f38ad177-3298-48f2-9ec0-18a4a438a664-maas-api-openshift-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:05.023487 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:05.023468    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38ad177-3298-48f2-9ec0-18a4a438a664-maas-api-xks-ca" pod "f38ad177-3298-48f2-9ec0-18a4a438a664" (UID: "f38ad177-3298-48f2-9ec0-18a4a438a664"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:05.023608 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:05.023588    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38ad177-3298-48f2-9ec0-18a4a438a664-kube-api-access-wxmlx" pod "f38ad177-3298-48f2-9ec0-18a4a438a664" (UID: "f38ad177-3298-48f2-9ec0-18a4a438a664"). InnerVolumeSpecName "kube-api-access-wxmlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:05.123370 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:05.123335    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxmlx\" (UniqueName: \"kubernetes.io/projected/f38ad177-3298-48f2-9ec0-18a4a438a664-kube-api-access-wxmlx\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:05.123370 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:05.123368    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f38ad177-3298-48f2-9ec0-18a4a438a664-maas-api-xks-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:05.771767 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:05.771739    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"
Oct 31 04:27:05.771971 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:05.771781    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv" event={"ID":"f38ad177-3298-48f2-9ec0-18a4a438a664","Type":"ContainerDied","Data":"acc195aa493434d75f72dd67186db65e96905211efc0f11940d7ed0f0457658b"}
Oct 31 04:27:05.771971 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:05.771841    4900 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acc195aa493434d75f72dd67186db65e96905211efc0f11940d7ed0f0457658b"
Oct 31 04:27:06.161531 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:06.161451    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:06.299072 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:06.299034    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"]
Oct 31 04:27:06.530377 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:06.530331    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:06.595418 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:06.595371    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6" podUID="8c101703-159d-4ae6-8e41-e669698c9130" containerName="payload-pre-processing" containerID="cri-o://12260331744c799f43c0b87b8144d6e63f1990a5780ef863dd204d917e3794bc" gracePeriod=25
Oct 31 04:27:06.984835 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:27:06.984798    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:27:07.103707 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:07.103660    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:08.749917 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:08.749882    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6" podUID="8c101703-159d-4ae6-8e41-e669698c9130" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:27:09.412924 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:09.412889    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:10.872243 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:10.872206    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:12.665812 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:12.665753    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:12.817246 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:12.817215    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"]
Oct 31 04:27:12.817431 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:12.817415    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj" podUID="36e17b74-45d2-4d36-842f-1c1868c501e0" containerName="istio-proxy" containerID="cri-o://30c186b1f373172b66a294b7ab7eeab8078cb5f5072ef4427ee129d396c24fac" gracePeriod=30
Oct 31 04:27:12.880687 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:12.880631    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:13.649002 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:13.648968    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:14.655802 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:14.655752    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t" podUID="48a65552-b7f7-421c-b714-3e3f2f2b3dc2" containerName="main" containerID="cri-o://d5d5ed9e778845eb0c008f6ff797dd7a6f75f8a16b9e1eb240a1fb526138b598" gracePeriod=45
Oct 31 04:27:14.822733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:14.822701    4900 generic.go:356] "Generic (PLEG): container finished" podID="48a65552-b7f7-421c-b714-3e3f2f2b3dc2" containerID="d5d5ed9e778845eb0c008f6ff797dd7a6f75f8a16b9e1eb240a1fb526138b598" exitCode=0
Oct 31 04:27:14.822944 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:14.822802    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t" event={"ID":"48a65552-b7f7-421c-b714-3e3f2f2b3dc2","Type":"ContainerDied","Data":"d5d5ed9e778845eb0c008f6ff797dd7a6f75f8a16b9e1eb240a1fb526138b598"}
Oct 31 04:27:14.826093 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:14.826065    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t" podUID="48a65552-b7f7-421c-b714-3e3f2f2b3dc2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.195:8000/ready\": dial tcp 10.133.0.195:8000: connect: connection refused"
Oct 31 04:27:14.916233 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:14.916174    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:27:15.050311 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.050278    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-tmp-dir\") pod \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") "
Oct 31 04:27:15.050484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.050351    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-kube-api-access-qnxgs\" (UniqueName: \"kubernetes.io/projected/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-kube-api-access-qnxgs\") pod \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") "
Oct 31 04:27:15.050484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.050401    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-tls-certs\" (UniqueName: \"kubernetes.io/secret/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-tls-certs\") pod \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") "
Oct 31 04:27:15.050484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.050441    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-model-cache\") pod \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") "
Oct 31 04:27:15.050699 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.050547    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-dshm\") pod \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") "
Oct 31 04:27:15.050699 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.050572    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-tmp-dir" pod "48a65552-b7f7-421c-b714-3e3f2f2b3dc2" (UID: "48a65552-b7f7-421c-b714-3e3f2f2b3dc2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:15.050699 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.050634    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-home\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-home\") pod \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\" (UID: \"48a65552-b7f7-421c-b714-3e3f2f2b3dc2\") "
Oct 31 04:27:15.050894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.050808    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-model-cache" pod "48a65552-b7f7-421c-b714-3e3f2f2b3dc2" (UID: "48a65552-b7f7-421c-b714-3e3f2f2b3dc2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:15.051013 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.050990    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-home" pod "48a65552-b7f7-421c-b714-3e3f2f2b3dc2" (UID: "48a65552-b7f7-421c-b714-3e3f2f2b3dc2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:15.051221 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.051202    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:15.051287 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.051227    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:15.051287 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.051241    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:15.053116 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.053091    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-dshm" pod "48a65552-b7f7-421c-b714-3e3f2f2b3dc2" (UID: "48a65552-b7f7-421c-b714-3e3f2f2b3dc2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:15.053299 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.053277    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-tls-certs" pod "48a65552-b7f7-421c-b714-3e3f2f2b3dc2" (UID: "48a65552-b7f7-421c-b714-3e3f2f2b3dc2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:15.053348 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.053297    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-kube-api-access-qnxgs" pod "48a65552-b7f7-421c-b714-3e3f2f2b3dc2" (UID: "48a65552-b7f7-421c-b714-3e3f2f2b3dc2"). InnerVolumeSpecName "kube-api-access-qnxgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:15.151867 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.151833    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnxgs\" (UniqueName: \"kubernetes.io/projected/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-kube-api-access-qnxgs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:15.151867 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.151863    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:15.151867 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.151873    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a65552-b7f7-421c-b714-3e3f2f2b3dc2-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:15.830618 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.830590    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"
Oct 31 04:27:15.831068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.830592    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t" event={"ID":"48a65552-b7f7-421c-b714-3e3f2f2b3dc2","Type":"ContainerDied","Data":"2df604d1c093ba16c1d6ebdc2b1d08ecd765a2ff0c87148005008ddfdd2a1756"}
Oct 31 04:27:15.831068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.830707    4900 scope.go:122] "RemoveContainer" containerID="d5d5ed9e778845eb0c008f6ff797dd7a6f75f8a16b9e1eb240a1fb526138b598"
Oct 31 04:27:15.996924 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:15.996891    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"]
Oct 31 04:27:16.046959 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:16.046926    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-502ec2/rate-test-model-502ec2-kserve-7d9847c485-8f29t"]
Oct 31 04:27:16.252421 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:16.252391    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:16.454768 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:16.454729    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:17.167883 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:17.167840    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a65552-b7f7-421c-b714-3e3f2f2b3dc2" path="/var/lib/kubelet/pods/48a65552-b7f7-421c-b714-3e3f2f2b3dc2/volumes"
Oct 31 04:27:18.096405 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.096029    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:27:18.284024 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.283983    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-envoy\") pod \"36e17b74-45d2-4d36-842f-1c1868c501e0\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") "
Oct 31 04:27:18.284540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.284093    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-workload-socket\") pod \"36e17b74-45d2-4d36-842f-1c1868c501e0\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") "
Oct 31 04:27:18.284540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.284133    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36e17b74-45d2-4d36-842f-1c1868c501e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e17b74-45d2-4d36-842f-1c1868c501e0-istiod-ca-cert\") pod \"36e17b74-45d2-4d36-842f-1c1868c501e0\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") "
Oct 31 04:27:18.284540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.284214    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-podinfo\") pod \"36e17b74-45d2-4d36-842f-1c1868c501e0\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") "
Oct 31 04:27:18.284540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.284287    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-token\" (UniqueName: \"kubernetes.io/projected/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-token\") pod \"36e17b74-45d2-4d36-842f-1c1868c501e0\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") "
Oct 31 04:27:18.284540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.284329    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e17b74-45d2-4d36-842f-1c1868c501e0-kube-api-access-679h8\" (UniqueName: \"kubernetes.io/projected/36e17b74-45d2-4d36-842f-1c1868c501e0-kube-api-access-679h8\") pod \"36e17b74-45d2-4d36-842f-1c1868c501e0\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") "
Oct 31 04:27:18.284540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.284386    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-credential-socket\") pod \"36e17b74-45d2-4d36-842f-1c1868c501e0\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") "
Oct 31 04:27:18.284540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.284406    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-workload-socket" pod "36e17b74-45d2-4d36-842f-1c1868c501e0" (UID: "36e17b74-45d2-4d36-842f-1c1868c501e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:18.284540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.284418    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-data\") pod \"36e17b74-45d2-4d36-842f-1c1868c501e0\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") "
Oct 31 04:27:18.284540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.284456    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-workload-certs\") pod \"36e17b74-45d2-4d36-842f-1c1868c501e0\" (UID: \"36e17b74-45d2-4d36-842f-1c1868c501e0\") "
Oct 31 04:27:18.285088 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.284661    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-credential-socket" pod "36e17b74-45d2-4d36-842f-1c1868c501e0" (UID: "36e17b74-45d2-4d36-842f-1c1868c501e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:18.285088 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.284687    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36e17b74-45d2-4d36-842f-1c1868c501e0-istiod-ca-cert" pod "36e17b74-45d2-4d36-842f-1c1868c501e0" (UID: "36e17b74-45d2-4d36-842f-1c1868c501e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:18.285088 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.284919    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-workload-certs" pod "36e17b74-45d2-4d36-842f-1c1868c501e0" (UID: "36e17b74-45d2-4d36-842f-1c1868c501e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:18.285088 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.284955    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-data" pod "36e17b74-45d2-4d36-842f-1c1868c501e0" (UID: "36e17b74-45d2-4d36-842f-1c1868c501e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:18.285340 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.285314    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:18.285403 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.285344    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:18.285403 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.285360    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:18.285403 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.285376    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e17b74-45d2-4d36-842f-1c1868c501e0-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:18.285403 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.285392    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:18.286589 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.286556    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-envoy" pod "36e17b74-45d2-4d36-842f-1c1868c501e0" (UID: "36e17b74-45d2-4d36-842f-1c1868c501e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:18.286918 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.286896    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e17b74-45d2-4d36-842f-1c1868c501e0-kube-api-access-679h8" pod "36e17b74-45d2-4d36-842f-1c1868c501e0" (UID: "36e17b74-45d2-4d36-842f-1c1868c501e0"). InnerVolumeSpecName "kube-api-access-679h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:18.287089 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.287071    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-token" pod "36e17b74-45d2-4d36-842f-1c1868c501e0" (UID: "36e17b74-45d2-4d36-842f-1c1868c501e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:18.287275 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.287257    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-podinfo" pod "36e17b74-45d2-4d36-842f-1c1868c501e0" (UID: "36e17b74-45d2-4d36-842f-1c1868c501e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:27:18.387269 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.387165    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:18.387269 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.387201    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-679h8\" (UniqueName: \"kubernetes.io/projected/36e17b74-45d2-4d36-842f-1c1868c501e0-kube-api-access-679h8\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:18.387269 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.387218    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:18.387269 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.387232    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e17b74-45d2-4d36-842f-1c1868c501e0-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:18.750364 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.750320    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6" podUID="8c101703-159d-4ae6-8e41-e669698c9130" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:27:18.851299 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.851265    4900 generic.go:356] "Generic (PLEG): container finished" podID="36e17b74-45d2-4d36-842f-1c1868c501e0" containerID="30c186b1f373172b66a294b7ab7eeab8078cb5f5072ef4427ee129d396c24fac" exitCode=0
Oct 31 04:27:18.851484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.851321    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"
Oct 31 04:27:18.851484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.851312    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj" event={"ID":"36e17b74-45d2-4d36-842f-1c1868c501e0","Type":"ContainerDied","Data":"30c186b1f373172b66a294b7ab7eeab8078cb5f5072ef4427ee129d396c24fac"}
Oct 31 04:27:18.851484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.851452    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj" event={"ID":"36e17b74-45d2-4d36-842f-1c1868c501e0","Type":"ContainerDied","Data":"ba2b6aae5069b7515e4616d8825c4fe3042ddb984e349763ee55e14645dc71bc"}
Oct 31 04:27:18.851484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.851480    4900 scope.go:122] "RemoveContainer" containerID="30c186b1f373172b66a294b7ab7eeab8078cb5f5072ef4427ee129d396c24fac"
Oct 31 04:27:18.861750 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.861728    4900 scope.go:122] "RemoveContainer" containerID="5f2f03666608e01fe6c38cf054d95fa8019df4b1e303defca2b7bf9ab7911e83"
Oct 31 04:27:18.870348 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.870330    4900 scope.go:122] "RemoveContainer" containerID="30c186b1f373172b66a294b7ab7eeab8078cb5f5072ef4427ee129d396c24fac"
Oct 31 04:27:18.870606 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:27:18.870587    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c186b1f373172b66a294b7ab7eeab8078cb5f5072ef4427ee129d396c24fac\": container with ID starting with 30c186b1f373172b66a294b7ab7eeab8078cb5f5072ef4427ee129d396c24fac not found: ID does not exist" containerID="30c186b1f373172b66a294b7ab7eeab8078cb5f5072ef4427ee129d396c24fac"
Oct 31 04:27:18.870663 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.870615    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c186b1f373172b66a294b7ab7eeab8078cb5f5072ef4427ee129d396c24fac"} err="failed to get container status \"30c186b1f373172b66a294b7ab7eeab8078cb5f5072ef4427ee129d396c24fac\": rpc error: code = NotFound desc = could not find container \"30c186b1f373172b66a294b7ab7eeab8078cb5f5072ef4427ee129d396c24fac\": container with ID starting with 30c186b1f373172b66a294b7ab7eeab8078cb5f5072ef4427ee129d396c24fac not found: ID does not exist"
Oct 31 04:27:18.870663 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.870635    4900 scope.go:122] "RemoveContainer" containerID="5f2f03666608e01fe6c38cf054d95fa8019df4b1e303defca2b7bf9ab7911e83"
Oct 31 04:27:18.870884 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:27:18.870867    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f2f03666608e01fe6c38cf054d95fa8019df4b1e303defca2b7bf9ab7911e83\": container with ID starting with 5f2f03666608e01fe6c38cf054d95fa8019df4b1e303defca2b7bf9ab7911e83 not found: ID does not exist" containerID="5f2f03666608e01fe6c38cf054d95fa8019df4b1e303defca2b7bf9ab7911e83"
Oct 31 04:27:18.870953 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:18.870893    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f2f03666608e01fe6c38cf054d95fa8019df4b1e303defca2b7bf9ab7911e83"} err="failed to get container status \"5f2f03666608e01fe6c38cf054d95fa8019df4b1e303defca2b7bf9ab7911e83\": rpc error: code = NotFound desc = could not find container \"5f2f03666608e01fe6c38cf054d95fa8019df4b1e303defca2b7bf9ab7911e83\": container with ID starting with 5f2f03666608e01fe6c38cf054d95fa8019df4b1e303defca2b7bf9ab7911e83 not found: ID does not exist"
Oct 31 04:27:19.016101 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:19.016014    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"]
Oct 31 04:27:19.066273 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:19.066243    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-26c74d32-openshift-default-5f887dbf77-pgkfj"]
Oct 31 04:27:19.159134 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:19.159100    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36e17b74-45d2-4d36-842f-1c1868c501e0" path="/var/lib/kubelet/pods/36e17b74-45d2-4d36-842f-1c1868c501e0/volumes"
Oct 31 04:27:20.408339 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:20.408301    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:21.047504 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.047465    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"]
Oct 31 04:27:21.047738 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.047695    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz" podUID="9d97d1f9-e5a1-4768-9940-f81b31df1f44" containerName="istio-proxy" containerID="cri-o://7f32167da6fef0eb1065ae58aad9615f0956ad43f4f35b860bcce4051860227d" gracePeriod=30
Oct 31 04:27:21.260690 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.260650    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:21.319486 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.319390    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerName="main" containerID="cri-o://d4e6e9c9e0a4351d46a04e64e9bdcbfb14a188ab1f9b5e3c2d535a740f2e518d" gracePeriod=45
Oct 31 04:27:21.577456 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.577376    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 04:27:21.622515 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.622476    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-dshm\") pod \"64487e0c-7344-45c3-af5e-23c2681cc64e\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") "
Oct 31 04:27:21.622515 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.622521    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64487e0c-7344-45c3-af5e-23c2681cc64e-tls-certs\" (UniqueName: \"kubernetes.io/secret/64487e0c-7344-45c3-af5e-23c2681cc64e-tls-certs\") pod \"64487e0c-7344-45c3-af5e-23c2681cc64e\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") "
Oct 31 04:27:21.622749 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.622583    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-model-cache\") pod \"64487e0c-7344-45c3-af5e-23c2681cc64e\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") "
Oct 31 04:27:21.622749 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.622600    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-tmp-dir\") pod \"64487e0c-7344-45c3-af5e-23c2681cc64e\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") "
Oct 31 04:27:21.622749 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.622633    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-home\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-home\") pod \"64487e0c-7344-45c3-af5e-23c2681cc64e\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") "
Oct 31 04:27:21.622749 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.622699    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64487e0c-7344-45c3-af5e-23c2681cc64e-kube-api-access-vdjjz\" (UniqueName: \"kubernetes.io/projected/64487e0c-7344-45c3-af5e-23c2681cc64e-kube-api-access-vdjjz\") pod \"64487e0c-7344-45c3-af5e-23c2681cc64e\" (UID: \"64487e0c-7344-45c3-af5e-23c2681cc64e\") "
Oct 31 04:27:21.623015 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.622905    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-model-cache" pod "64487e0c-7344-45c3-af5e-23c2681cc64e" (UID: "64487e0c-7344-45c3-af5e-23c2681cc64e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.623015 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.622918    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-tmp-dir" pod "64487e0c-7344-45c3-af5e-23c2681cc64e" (UID: "64487e0c-7344-45c3-af5e-23c2681cc64e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.623015 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.622975    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-home" pod "64487e0c-7344-45c3-af5e-23c2681cc64e" (UID: "64487e0c-7344-45c3-af5e-23c2681cc64e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.623130 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.623121    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.623193 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.623134    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.623193 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.623144    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.624737 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.624714    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-dshm" pod "64487e0c-7344-45c3-af5e-23c2681cc64e" (UID: "64487e0c-7344-45c3-af5e-23c2681cc64e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:21.624737 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.624723    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64487e0c-7344-45c3-af5e-23c2681cc64e-tls-certs" pod "64487e0c-7344-45c3-af5e-23c2681cc64e" (UID: "64487e0c-7344-45c3-af5e-23c2681cc64e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:27:21.624914 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.624776    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64487e0c-7344-45c3-af5e-23c2681cc64e-kube-api-access-vdjjz" pod "64487e0c-7344-45c3-af5e-23c2681cc64e" (UID: "64487e0c-7344-45c3-af5e-23c2681cc64e"). InnerVolumeSpecName "kube-api-access-vdjjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:21.723883 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.723848    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64487e0c-7344-45c3-af5e-23c2681cc64e-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.723883 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.723880    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64487e0c-7344-45c3-af5e-23c2681cc64e-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.724088 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.723892    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdjjz\" (UniqueName: \"kubernetes.io/projected/64487e0c-7344-45c3-af5e-23c2681cc64e-kube-api-access-vdjjz\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:21.872195 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.872101    4900 generic.go:356] "Generic (PLEG): container finished" podID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerID="d4e6e9c9e0a4351d46a04e64e9bdcbfb14a188ab1f9b5e3c2d535a740f2e518d" exitCode=0
Oct 31 04:27:21.872195 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.872161    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"
Oct 31 04:27:21.872411 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.872209    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" event={"ID":"64487e0c-7344-45c3-af5e-23c2681cc64e","Type":"ContainerDied","Data":"d4e6e9c9e0a4351d46a04e64e9bdcbfb14a188ab1f9b5e3c2d535a740f2e518d"}
Oct 31 04:27:21.872411 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.872262    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z" event={"ID":"64487e0c-7344-45c3-af5e-23c2681cc64e","Type":"ContainerDied","Data":"be31c8b1cca59bb63496a5d37742ed176b90a0c9cccd1e9d079f8e4db1290850"}
Oct 31 04:27:21.872411 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.872285    4900 scope.go:122] "RemoveContainer" containerID="d4e6e9c9e0a4351d46a04e64e9bdcbfb14a188ab1f9b5e3c2d535a740f2e518d"
Oct 31 04:27:21.883067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.883041    4900 scope.go:122] "RemoveContainer" containerID="d4e6e9c9e0a4351d46a04e64e9bdcbfb14a188ab1f9b5e3c2d535a740f2e518d"
Oct 31 04:27:21.883388 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:27:21.883360    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e6e9c9e0a4351d46a04e64e9bdcbfb14a188ab1f9b5e3c2d535a740f2e518d\": container with ID starting with d4e6e9c9e0a4351d46a04e64e9bdcbfb14a188ab1f9b5e3c2d535a740f2e518d not found: ID does not exist" containerID="d4e6e9c9e0a4351d46a04e64e9bdcbfb14a188ab1f9b5e3c2d535a740f2e518d"
Oct 31 04:27:21.883466 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.883400    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e6e9c9e0a4351d46a04e64e9bdcbfb14a188ab1f9b5e3c2d535a740f2e518d"} err="failed to get container status \"d4e6e9c9e0a4351d46a04e64e9bdcbfb14a188ab1f9b5e3c2d535a740f2e518d\": rpc error: code = NotFound desc = could not find container \"d4e6e9c9e0a4351d46a04e64e9bdcbfb14a188ab1f9b5e3c2d535a740f2e518d\": container with ID starting with d4e6e9c9e0a4351d46a04e64e9bdcbfb14a188ab1f9b5e3c2d535a740f2e518d not found: ID does not exist"
Oct 31 04:27:21.924108 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:21.924075    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:22.035290 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:22.035245    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"]
Oct 31 04:27:22.038632 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:22.038603    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d39257/distinct-w1-d39257-kserve-7b49987986-vbv8z"]
Oct 31 04:27:22.135647 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:22.135562    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:23.151625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:23.151587    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" path="/var/lib/kubelet/pods/64487e0c-7344-45c3-af5e-23c2681cc64e/volumes"
Oct 31 04:27:23.348332 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:23.348299    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:23.984272 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:23.984237    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"]
Oct 31 04:27:24.094452 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:24.094415    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"]
Oct 31 04:27:24.429671 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:24.429581    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:24.629247 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:24.629207    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:26.303582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.303555    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:27:26.380591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.380497    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-credential-socket\") pod \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") "
Oct 31 04:27:26.380591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.380551    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-workload-certs\") pod \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") "
Oct 31 04:27:26.380591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.380580    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-data\") pod \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") "
Oct 31 04:27:26.380961 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.380611    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d97d1f9-e5a1-4768-9940-f81b31df1f44-kube-api-access-qvq7h\" (UniqueName: \"kubernetes.io/projected/9d97d1f9-e5a1-4768-9940-f81b31df1f44-kube-api-access-qvq7h\") pod \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") "
Oct 31 04:27:26.380961 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.380647    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-envoy\") pod \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") "
Oct 31 04:27:26.380961 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.380718    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-podinfo\") pod \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") "
Oct 31 04:27:26.380961 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.380865    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istiod-ca-cert\") pod \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") "
Oct 31 04:27:26.380961 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.380855    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-credential-socket" pod "9d97d1f9-e5a1-4768-9940-f81b31df1f44" (UID: "9d97d1f9-e5a1-4768-9940-f81b31df1f44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:26.380961 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.380903    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-data" pod "9d97d1f9-e5a1-4768-9940-f81b31df1f44" (UID: "9d97d1f9-e5a1-4768-9940-f81b31df1f44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:26.380961 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.380944    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-workload-socket\") pod \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") "
Oct 31 04:27:26.381327 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.381005    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-token\" (UniqueName: \"kubernetes.io/projected/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-token\") pod \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\" (UID: \"9d97d1f9-e5a1-4768-9940-f81b31df1f44\") "
Oct 31 04:27:26.381327 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.381023    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-workload-certs" pod "9d97d1f9-e5a1-4768-9940-f81b31df1f44" (UID: "9d97d1f9-e5a1-4768-9940-f81b31df1f44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:26.381327 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.381118    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-workload-socket" pod "9d97d1f9-e5a1-4768-9940-f81b31df1f44" (UID: "9d97d1f9-e5a1-4768-9940-f81b31df1f44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:26.381327 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.381257    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istiod-ca-cert" pod "9d97d1f9-e5a1-4768-9940-f81b31df1f44" (UID: "9d97d1f9-e5a1-4768-9940-f81b31df1f44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:26.381538 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.381444    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.381538 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.381466    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.381538 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.381479    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.381538 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.381493    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.381538 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.381503    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.383194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.383164    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d97d1f9-e5a1-4768-9940-f81b31df1f44-kube-api-access-qvq7h" pod "9d97d1f9-e5a1-4768-9940-f81b31df1f44" (UID: "9d97d1f9-e5a1-4768-9940-f81b31df1f44"). InnerVolumeSpecName "kube-api-access-qvq7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:26.383309 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.383227    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-token" pod "9d97d1f9-e5a1-4768-9940-f81b31df1f44" (UID: "9d97d1f9-e5a1-4768-9940-f81b31df1f44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:26.383479 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.383454    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-envoy" pod "9d97d1f9-e5a1-4768-9940-f81b31df1f44" (UID: "9d97d1f9-e5a1-4768-9940-f81b31df1f44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:26.383586 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.383500    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-podinfo" pod "9d97d1f9-e5a1-4768-9940-f81b31df1f44" (UID: "9d97d1f9-e5a1-4768-9940-f81b31df1f44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:27:26.482543 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.482506    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvq7h\" (UniqueName: \"kubernetes.io/projected/9d97d1f9-e5a1-4768-9940-f81b31df1f44-kube-api-access-qvq7h\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.482543 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.482541    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.482543 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.482552    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.482800 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.482561    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d97d1f9-e5a1-4768-9940-f81b31df1f44-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.686193 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.686110    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:26.833782 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.833753    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"
Oct 31 04:27:26.888054 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.888020    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c101703-159d-4ae6-8e41-e669698c9130-kube-api-access-gt9pj\" (UniqueName: \"kubernetes.io/projected/8c101703-159d-4ae6-8e41-e669698c9130-kube-api-access-gt9pj\") pod \"8c101703-159d-4ae6-8e41-e669698c9130\" (UID: \"8c101703-159d-4ae6-8e41-e669698c9130\") "
Oct 31 04:27:26.888243 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.888166    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c101703-159d-4ae6-8e41-e669698c9130-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c101703-159d-4ae6-8e41-e669698c9130-plugins-config-volume\") pod \"8c101703-159d-4ae6-8e41-e669698c9130\" (UID: \"8c101703-159d-4ae6-8e41-e669698c9130\") "
Oct 31 04:27:26.888525 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.888502    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c101703-159d-4ae6-8e41-e669698c9130-plugins-config-volume" pod "8c101703-159d-4ae6-8e41-e669698c9130" (UID: "8c101703-159d-4ae6-8e41-e669698c9130"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:26.890147 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.890124    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c101703-159d-4ae6-8e41-e669698c9130-kube-api-access-gt9pj" pod "8c101703-159d-4ae6-8e41-e669698c9130" (UID: "8c101703-159d-4ae6-8e41-e669698c9130"). InnerVolumeSpecName "kube-api-access-gt9pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:26.902263 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.902236    4900 generic.go:356] "Generic (PLEG): container finished" podID="9d97d1f9-e5a1-4768-9940-f81b31df1f44" containerID="7f32167da6fef0eb1065ae58aad9615f0956ad43f4f35b860bcce4051860227d" exitCode=0
Oct 31 04:27:26.902436 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.902289    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"
Oct 31 04:27:26.902436 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.902320    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz" event={"ID":"9d97d1f9-e5a1-4768-9940-f81b31df1f44","Type":"ContainerDied","Data":"7f32167da6fef0eb1065ae58aad9615f0956ad43f4f35b860bcce4051860227d"}
Oct 31 04:27:26.902436 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.902375    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz" event={"ID":"9d97d1f9-e5a1-4768-9940-f81b31df1f44","Type":"ContainerDied","Data":"10185bcb221a22a039951c0dada3709ad637f94848e0a08c5c3ea11d8144315b"}
Oct 31 04:27:26.902436 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.902406    4900 scope.go:122] "RemoveContainer" containerID="7f32167da6fef0eb1065ae58aad9615f0956ad43f4f35b860bcce4051860227d"
Oct 31 04:27:26.903666 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.903640    4900 generic.go:356] "Generic (PLEG): container finished" podID="8c101703-159d-4ae6-8e41-e669698c9130" containerID="12260331744c799f43c0b87b8144d6e63f1990a5780ef863dd204d917e3794bc" exitCode=0
Oct 31 04:27:26.903821 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.903687    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"
Oct 31 04:27:26.903821 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.903724    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6" event={"ID":"8c101703-159d-4ae6-8e41-e669698c9130","Type":"ContainerDied","Data":"12260331744c799f43c0b87b8144d6e63f1990a5780ef863dd204d917e3794bc"}
Oct 31 04:27:26.903821 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.903764    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6" event={"ID":"8c101703-159d-4ae6-8e41-e669698c9130","Type":"ContainerDied","Data":"af3e603eae7e165d913de69c894085ce62bd2198c031ecbedce109b991c209bf"}
Oct 31 04:27:26.913889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.912960    4900 scope.go:122] "RemoveContainer" containerID="7f32167da6fef0eb1065ae58aad9615f0956ad43f4f35b860bcce4051860227d"
Oct 31 04:27:26.913889 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:27:26.913275    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f32167da6fef0eb1065ae58aad9615f0956ad43f4f35b860bcce4051860227d\": container with ID starting with 7f32167da6fef0eb1065ae58aad9615f0956ad43f4f35b860bcce4051860227d not found: ID does not exist" containerID="7f32167da6fef0eb1065ae58aad9615f0956ad43f4f35b860bcce4051860227d"
Oct 31 04:27:26.913889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.913301    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f32167da6fef0eb1065ae58aad9615f0956ad43f4f35b860bcce4051860227d"} err="failed to get container status \"7f32167da6fef0eb1065ae58aad9615f0956ad43f4f35b860bcce4051860227d\": rpc error: code = NotFound desc = could not find container \"7f32167da6fef0eb1065ae58aad9615f0956ad43f4f35b860bcce4051860227d\": container with ID starting with 7f32167da6fef0eb1065ae58aad9615f0956ad43f4f35b860bcce4051860227d not found: ID does not exist"
Oct 31 04:27:26.913889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.913319    4900 scope.go:122] "RemoveContainer" containerID="12260331744c799f43c0b87b8144d6e63f1990a5780ef863dd204d917e3794bc"
Oct 31 04:27:26.924245 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.924221    4900 scope.go:122] "RemoveContainer" containerID="12260331744c799f43c0b87b8144d6e63f1990a5780ef863dd204d917e3794bc"
Oct 31 04:27:26.924563 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:27:26.924537    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12260331744c799f43c0b87b8144d6e63f1990a5780ef863dd204d917e3794bc\": container with ID starting with 12260331744c799f43c0b87b8144d6e63f1990a5780ef863dd204d917e3794bc not found: ID does not exist" containerID="12260331744c799f43c0b87b8144d6e63f1990a5780ef863dd204d917e3794bc"
Oct 31 04:27:26.924650 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.924571    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12260331744c799f43c0b87b8144d6e63f1990a5780ef863dd204d917e3794bc"} err="failed to get container status \"12260331744c799f43c0b87b8144d6e63f1990a5780ef863dd204d917e3794bc\": rpc error: code = NotFound desc = could not find container \"12260331744c799f43c0b87b8144d6e63f1990a5780ef863dd204d917e3794bc\": container with ID starting with 12260331744c799f43c0b87b8144d6e63f1990a5780ef863dd204d917e3794bc not found: ID does not exist"
Oct 31 04:27:26.989399 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.989367    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c101703-159d-4ae6-8e41-e669698c9130-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:26.989399 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:26.989399    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt9pj\" (UniqueName: \"kubernetes.io/projected/8c101703-159d-4ae6-8e41-e669698c9130-kube-api-access-gt9pj\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:27.060150 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:27.060102    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"]
Oct 31 04:27:27.110044 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:27.110012    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-gl6cz"]
Oct 31 04:27:27.153484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:27.153448    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d97d1f9-e5a1-4768-9940-f81b31df1f44" path="/var/lib/kubelet/pods/9d97d1f9-e5a1-4768-9940-f81b31df1f44/volumes"
Oct 31 04:27:27.170928 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:27.170893    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"]
Oct 31 04:27:27.175061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:27.175033    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-2339ed-56d678b89f-fppm6"]
Oct 31 04:27:28.979642 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:28.979612    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:27:28.984644 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:28.984612    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng" podUID="0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7" containerName="payload-processing" containerID="cri-o://9635719a59d69ae4f3581c6428f26a953b8b636660a8cb2e96938ae3fa02c5fc" gracePeriod=25
Oct 31 04:27:28.985253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:28.985232    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:27:29.094901 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:29.094859    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6" podUID="c0b7639f-ab0a-4634-9c22-c6408daeb5d0" containerName="payload-pre-processing" containerID="cri-o://5ec7b7d965b540dfa1dc602036ed07b8fda38965cff22f44c90b709e9205506d" gracePeriod=25
Oct 31 04:27:29.155263 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:29.155222    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c101703-159d-4ae6-8e41-e669698c9130" path="/var/lib/kubelet/pods/8c101703-159d-4ae6-8e41-e669698c9130/volumes"
Oct 31 04:27:29.237423 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:29.237393    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:31.463437 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:31.463403    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:32.778880 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:32.778842    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6" podUID="c0b7639f-ab0a-4634-9c22-c6408daeb5d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:27:33.785686 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:33.785644    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng" podUID="0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:27:35.660016 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:35.659985    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:35.861559 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:35.861530    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:36.064847 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:36.064812    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:36.839537 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:36.839493    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:36.983316 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:27:36.983280    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:27:38.674265 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:38.674222    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:39.052904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:39.052866    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"]
Oct 31 04:27:39.102616 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:39.102579    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40561656-clhld"]
Oct 31 04:27:39.152750 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:39.152717    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3113317-83f1-480e-8522-dba71d8bb657" path="/var/lib/kubelet/pods/b3113317-83f1-480e-8522-dba71d8bb657/volumes"
Oct 31 04:27:42.066060 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:42.066012    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:42.285046 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:42.285009    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:42.779589 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:42.779547    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6" podUID="c0b7639f-ab0a-4634-9c22-c6408daeb5d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:27:43.537746 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:43.537712    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:43.753519 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:43.753483    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:43.784986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:43.784953    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng" podUID="0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:27:44.040911 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:44.040749    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"]
Oct 31 04:27:44.041108 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:44.041032    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt" podUID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerName="istio-proxy" containerID="cri-o://7949a4a247db97bf76ddc6aea91b560f0697e951cc38f71a272a7c619d9c6f1f" gracePeriod=30
Oct 31 04:27:44.125274 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:44.125233    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:44.339032 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:44.338941    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:44.549828 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:44.549783    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:45.065204 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:45.065169    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:45.279721 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:45.279683    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:45.448700 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:45.448596    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"]
Oct 31 04:27:45.448935 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:45.448907    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8" podUID="97819a0a-0328-4845-b5ca-c25af6385a11" containerName="istio-proxy" containerID="cri-o://5e72147ab15ee0d2590fa42f6238dbbf274746ca75d3a0db8b9d5f07cc4e27d8" gracePeriod=30
Oct 31 04:27:45.536505 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:45.536466    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:46.306181 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:46.306143    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:48.841836 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:48.841797    4900 patch_prober.go:29] interesting pod/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 04:27:48.842345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:48.841861    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt" podUID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 04:27:49.103274 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.103194    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:49.307464 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.307440    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"
Oct 31 04:27:49.374563 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.374539    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 04:27:49.382770 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.382748    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"
Oct 31 04:27:49.446461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.446427    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7-plugins-config-volume\") pod \"0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7\" (UID: \"0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7\") "
Oct 31 04:27:49.446625 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.446521    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7-kube-api-access-l4rgt\" (UniqueName: \"kubernetes.io/projected/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7-kube-api-access-l4rgt\") pod \"0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7\" (UID: \"0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7\") "
Oct 31 04:27:49.446799 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.446766    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7-plugins-config-volume" pod "0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7" (UID: "0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:49.447162 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.447141    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:49.448606 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.448587    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7-kube-api-access-l4rgt" pod "0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7" (UID: "0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7"). InnerVolumeSpecName "kube-api-access-l4rgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:49.548056 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548025    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-podinfo\") pod \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") "
Oct 31 04:27:49.548256 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548076    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-token\" (UniqueName: \"kubernetes.io/projected/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-token\") pod \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") "
Oct 31 04:27:49.548256 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548111    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b7639f-ab0a-4634-9c22-c6408daeb5d0-kube-api-access-cbcpf\" (UniqueName: \"kubernetes.io/projected/c0b7639f-ab0a-4634-9c22-c6408daeb5d0-kube-api-access-cbcpf\") pod \"c0b7639f-ab0a-4634-9c22-c6408daeb5d0\" (UID: \"c0b7639f-ab0a-4634-9c22-c6408daeb5d0\") "
Oct 31 04:27:49.548256 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548156    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-data\") pod \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") "
Oct 31 04:27:49.548256 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548182    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-envoy\") pod \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") "
Oct 31 04:27:49.548256 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548212    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-credential-socket\") pod \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") "
Oct 31 04:27:49.548547 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548262    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-workload-socket\") pod \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") "
Oct 31 04:27:49.548547 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548303    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istiod-ca-cert\") pod \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") "
Oct 31 04:27:49.548547 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548399    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-kube-api-access-bkw7g\" (UniqueName: \"kubernetes.io/projected/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-kube-api-access-bkw7g\") pod \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") "
Oct 31 04:27:49.548547 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548448    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b7639f-ab0a-4634-9c22-c6408daeb5d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b7639f-ab0a-4634-9c22-c6408daeb5d0-plugins-config-volume\") pod \"c0b7639f-ab0a-4634-9c22-c6408daeb5d0\" (UID: \"c0b7639f-ab0a-4634-9c22-c6408daeb5d0\") "
Oct 31 04:27:49.548547 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548513    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-workload-certs\") pod \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\" (UID: \"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2\") "
Oct 31 04:27:49.548895 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548642    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-credential-socket" pod "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" (UID: "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:49.548895 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548862    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-workload-socket" pod "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" (UID: "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:49.548895 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548871    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-workload-certs" pod "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" (UID: "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:49.549116 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.549093    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:49.549184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.549124    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:49.549184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.549141    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4rgt\" (UniqueName: \"kubernetes.io/projected/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7-kube-api-access-l4rgt\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:49.549184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.549157    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:49.549823 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.548577    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-data" pod "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" (UID: "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:49.549823 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.549622    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istiod-ca-cert" pod "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" (UID: "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:49.549823 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.549737    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b7639f-ab0a-4634-9c22-c6408daeb5d0-plugins-config-volume" pod "c0b7639f-ab0a-4634-9c22-c6408daeb5d0" (UID: "c0b7639f-ab0a-4634-9c22-c6408daeb5d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:49.551028 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.550979    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-podinfo" pod "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" (UID: "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:27:49.551028 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.550992    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b7639f-ab0a-4634-9c22-c6408daeb5d0-kube-api-access-cbcpf" pod "c0b7639f-ab0a-4634-9c22-c6408daeb5d0" (UID: "c0b7639f-ab0a-4634-9c22-c6408daeb5d0"). InnerVolumeSpecName "kube-api-access-cbcpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:49.551028 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.550981    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-token" pod "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" (UID: "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:49.551287 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.551270    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-kube-api-access-bkw7g" pod "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" (UID: "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2"). InnerVolumeSpecName "kube-api-access-bkw7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:49.551370 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.551350    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-envoy" pod "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" (UID: "82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:49.650458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.650334    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:49.650458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.650367    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:49.650458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.650379    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbcpf\" (UniqueName: \"kubernetes.io/projected/c0b7639f-ab0a-4634-9c22-c6408daeb5d0-kube-api-access-cbcpf\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:49.650458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.650388    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:49.650458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.650399    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:49.650458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.650408    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:49.650458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.650416    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkw7g\" (UniqueName: \"kubernetes.io/projected/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2-kube-api-access-bkw7g\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:49.650458 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:49.650427    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0b7639f-ab0a-4634-9c22-c6408daeb5d0-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:50.031164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.031131    4900 generic.go:356] "Generic (PLEG): container finished" podID="0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7" containerID="9635719a59d69ae4f3581c6428f26a953b8b636660a8cb2e96938ae3fa02c5fc" exitCode=0
Oct 31 04:27:50.031596 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.031177    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"
Oct 31 04:27:50.031596 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.031212    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng" event={"ID":"0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7","Type":"ContainerDied","Data":"9635719a59d69ae4f3581c6428f26a953b8b636660a8cb2e96938ae3fa02c5fc"}
Oct 31 04:27:50.031596 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.031270    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng" event={"ID":"0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7","Type":"ContainerDied","Data":"1d8b1eb2e417a5324538872a4f068f76d9d0fb037e2b6d26b0d2b5979fda35fa"}
Oct 31 04:27:50.031596 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.031300    4900 scope.go:122] "RemoveContainer" containerID="9635719a59d69ae4f3581c6428f26a953b8b636660a8cb2e96938ae3fa02c5fc"
Oct 31 04:27:50.033077 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.033056    4900 generic.go:356] "Generic (PLEG): container finished" podID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerID="7949a4a247db97bf76ddc6aea91b560f0697e951cc38f71a272a7c619d9c6f1f" exitCode=0
Oct 31 04:27:50.033208 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.033105    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"
Oct 31 04:27:50.033208 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.033187    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt" event={"ID":"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2","Type":"ContainerDied","Data":"7949a4a247db97bf76ddc6aea91b560f0697e951cc38f71a272a7c619d9c6f1f"}
Oct 31 04:27:50.033316 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.033242    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt" event={"ID":"82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2","Type":"ContainerDied","Data":"64e7e31dbab15930d67f38537a106bbc402e83f0c5f591a64ce8e8d8a2ef5e97"}
Oct 31 04:27:50.034662 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.034639    4900 generic.go:356] "Generic (PLEG): container finished" podID="c0b7639f-ab0a-4634-9c22-c6408daeb5d0" containerID="5ec7b7d965b540dfa1dc602036ed07b8fda38965cff22f44c90b709e9205506d" exitCode=0
Oct 31 04:27:50.034797 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.034679    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"
Oct 31 04:27:50.034797 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.034686    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6" event={"ID":"c0b7639f-ab0a-4634-9c22-c6408daeb5d0","Type":"ContainerDied","Data":"5ec7b7d965b540dfa1dc602036ed07b8fda38965cff22f44c90b709e9205506d"}
Oct 31 04:27:50.034797 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.034722    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6" event={"ID":"c0b7639f-ab0a-4634-9c22-c6408daeb5d0","Type":"ContainerDied","Data":"757a23c20db6ee72ac9b94329da2ab90306c0a1d57ec65d3f9241e8f41819ed6"}
Oct 31 04:27:50.042563 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.042547    4900 scope.go:122] "RemoveContainer" containerID="9635719a59d69ae4f3581c6428f26a953b8b636660a8cb2e96938ae3fa02c5fc"
Oct 31 04:27:50.043144 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:27:50.043112    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9635719a59d69ae4f3581c6428f26a953b8b636660a8cb2e96938ae3fa02c5fc\": container with ID starting with 9635719a59d69ae4f3581c6428f26a953b8b636660a8cb2e96938ae3fa02c5fc not found: ID does not exist" containerID="9635719a59d69ae4f3581c6428f26a953b8b636660a8cb2e96938ae3fa02c5fc"
Oct 31 04:27:50.043259 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.043151    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9635719a59d69ae4f3581c6428f26a953b8b636660a8cb2e96938ae3fa02c5fc"} err="failed to get container status \"9635719a59d69ae4f3581c6428f26a953b8b636660a8cb2e96938ae3fa02c5fc\": rpc error: code = NotFound desc = could not find container \"9635719a59d69ae4f3581c6428f26a953b8b636660a8cb2e96938ae3fa02c5fc\": container with ID starting with 9635719a59d69ae4f3581c6428f26a953b8b636660a8cb2e96938ae3fa02c5fc not found: ID does not exist"
Oct 31 04:27:50.043259 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.043177    4900 scope.go:122] "RemoveContainer" containerID="7949a4a247db97bf76ddc6aea91b560f0697e951cc38f71a272a7c619d9c6f1f"
Oct 31 04:27:50.045811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.045767    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"]
Oct 31 04:27:50.048410 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.048391    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dc10fc5-nx5dd"]
Oct 31 04:27:50.052351 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.052337    4900 scope.go:122] "RemoveContainer" containerID="6af4eb84d7cb6c01d01e23a798aec50ffdf0df17716261749a2349c673e8f0d2"
Oct 31 04:27:50.062052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.062030    4900 scope.go:122] "RemoveContainer" containerID="7949a4a247db97bf76ddc6aea91b560f0697e951cc38f71a272a7c619d9c6f1f"
Oct 31 04:27:50.062354 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:27:50.062335    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7949a4a247db97bf76ddc6aea91b560f0697e951cc38f71a272a7c619d9c6f1f\": container with ID starting with 7949a4a247db97bf76ddc6aea91b560f0697e951cc38f71a272a7c619d9c6f1f not found: ID does not exist" containerID="7949a4a247db97bf76ddc6aea91b560f0697e951cc38f71a272a7c619d9c6f1f"
Oct 31 04:27:50.062413 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.062364    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7949a4a247db97bf76ddc6aea91b560f0697e951cc38f71a272a7c619d9c6f1f"} err="failed to get container status \"7949a4a247db97bf76ddc6aea91b560f0697e951cc38f71a272a7c619d9c6f1f\": rpc error: code = NotFound desc = could not find container \"7949a4a247db97bf76ddc6aea91b560f0697e951cc38f71a272a7c619d9c6f1f\": container with ID starting with 7949a4a247db97bf76ddc6aea91b560f0697e951cc38f71a272a7c619d9c6f1f not found: ID does not exist"
Oct 31 04:27:50.062413 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.062392    4900 scope.go:122] "RemoveContainer" containerID="6af4eb84d7cb6c01d01e23a798aec50ffdf0df17716261749a2349c673e8f0d2"
Oct 31 04:27:50.062641 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:27:50.062622    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af4eb84d7cb6c01d01e23a798aec50ffdf0df17716261749a2349c673e8f0d2\": container with ID starting with 6af4eb84d7cb6c01d01e23a798aec50ffdf0df17716261749a2349c673e8f0d2 not found: ID does not exist" containerID="6af4eb84d7cb6c01d01e23a798aec50ffdf0df17716261749a2349c673e8f0d2"
Oct 31 04:27:50.062710 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.062651    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af4eb84d7cb6c01d01e23a798aec50ffdf0df17716261749a2349c673e8f0d2"} err="failed to get container status \"6af4eb84d7cb6c01d01e23a798aec50ffdf0df17716261749a2349c673e8f0d2\": rpc error: code = NotFound desc = could not find container \"6af4eb84d7cb6c01d01e23a798aec50ffdf0df17716261749a2349c673e8f0d2\": container with ID starting with 6af4eb84d7cb6c01d01e23a798aec50ffdf0df17716261749a2349c673e8f0d2 not found: ID does not exist"
Oct 31 04:27:50.062710 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.062674    4900 scope.go:122] "RemoveContainer" containerID="5ec7b7d965b540dfa1dc602036ed07b8fda38965cff22f44c90b709e9205506d"
Oct 31 04:27:50.070848 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.070830    4900 scope.go:122] "RemoveContainer" containerID="5ec7b7d965b540dfa1dc602036ed07b8fda38965cff22f44c90b709e9205506d"
Oct 31 04:27:50.071109 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:27:50.071088    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec7b7d965b540dfa1dc602036ed07b8fda38965cff22f44c90b709e9205506d\": container with ID starting with 5ec7b7d965b540dfa1dc602036ed07b8fda38965cff22f44c90b709e9205506d not found: ID does not exist" containerID="5ec7b7d965b540dfa1dc602036ed07b8fda38965cff22f44c90b709e9205506d"
Oct 31 04:27:50.071150 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.071116    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec7b7d965b540dfa1dc602036ed07b8fda38965cff22f44c90b709e9205506d"} err="failed to get container status \"5ec7b7d965b540dfa1dc602036ed07b8fda38965cff22f44c90b709e9205506d\": rpc error: code = NotFound desc = could not find container \"5ec7b7d965b540dfa1dc602036ed07b8fda38965cff22f44c90b709e9205506d\": container with ID starting with 5ec7b7d965b540dfa1dc602036ed07b8fda38965cff22f44c90b709e9205506d not found: ID does not exist"
Oct 31 04:27:50.197218 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.197184    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"]
Oct 31 04:27:50.199986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.199954    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-93f411-67468fb94c-rrbn6"]
Oct 31 04:27:50.309419 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.309326    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"]
Oct 31 04:27:50.312624 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.312598    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-93f411-openshift-default-554df87979-6cgxt"]
Oct 31 04:27:50.421573 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.421539    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"]
Oct 31 04:27:50.425321 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.425293    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-93f411-8549bb767c-7zxng"]
Oct 31 04:27:50.688151 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.688125    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:50.700902 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.700870    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 04:27:50.760555 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.760522    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-istio-envoy\") pod \"97819a0a-0328-4845-b5ca-c25af6385a11\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") "
Oct 31 04:27:50.760765 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.760572    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-credential-socket\") pod \"97819a0a-0328-4845-b5ca-c25af6385a11\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") "
Oct 31 04:27:50.760765 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.760619    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97819a0a-0328-4845-b5ca-c25af6385a11-kube-api-access-xrctp\" (UniqueName: \"kubernetes.io/projected/97819a0a-0328-4845-b5ca-c25af6385a11-kube-api-access-xrctp\") pod \"97819a0a-0328-4845-b5ca-c25af6385a11\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") "
Oct 31 04:27:50.760765 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.760657    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-workload-certs\") pod \"97819a0a-0328-4845-b5ca-c25af6385a11\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") "
Oct 31 04:27:50.760765 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.760725    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-workload-socket\") pod \"97819a0a-0328-4845-b5ca-c25af6385a11\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") "
Oct 31 04:27:50.760765 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.760754    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97819a0a-0328-4845-b5ca-c25af6385a11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97819a0a-0328-4845-b5ca-c25af6385a11-istio-podinfo\") pod \"97819a0a-0328-4845-b5ca-c25af6385a11\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") "
Oct 31 04:27:50.761092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.760820    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97819a0a-0328-4845-b5ca-c25af6385a11-istio-token\" (UniqueName: \"kubernetes.io/projected/97819a0a-0328-4845-b5ca-c25af6385a11-istio-token\") pod \"97819a0a-0328-4845-b5ca-c25af6385a11\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") "
Oct 31 04:27:50.761092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.760875    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97819a0a-0328-4845-b5ca-c25af6385a11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97819a0a-0328-4845-b5ca-c25af6385a11-istiod-ca-cert\") pod \"97819a0a-0328-4845-b5ca-c25af6385a11\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") "
Oct 31 04:27:50.761092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.760906    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-istio-data\") pod \"97819a0a-0328-4845-b5ca-c25af6385a11\" (UID: \"97819a0a-0328-4845-b5ca-c25af6385a11\") "
Oct 31 04:27:50.761092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.760983    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-credential-socket" pod "97819a0a-0328-4845-b5ca-c25af6385a11" (UID: "97819a0a-0328-4845-b5ca-c25af6385a11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:50.761092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.761017    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-workload-certs" pod "97819a0a-0328-4845-b5ca-c25af6385a11" (UID: "97819a0a-0328-4845-b5ca-c25af6385a11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:50.761092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.761050    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-workload-socket" pod "97819a0a-0328-4845-b5ca-c25af6385a11" (UID: "97819a0a-0328-4845-b5ca-c25af6385a11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:50.761398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.761350    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-istio-data" pod "97819a0a-0328-4845-b5ca-c25af6385a11" (UID: "97819a0a-0328-4845-b5ca-c25af6385a11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:50.761398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.761364    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:50.761398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.761385    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:50.761530 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.761399    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:50.761530 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.761396    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97819a0a-0328-4845-b5ca-c25af6385a11-istiod-ca-cert" pod "97819a0a-0328-4845-b5ca-c25af6385a11" (UID: "97819a0a-0328-4845-b5ca-c25af6385a11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:27:50.763061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.763039    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97819a0a-0328-4845-b5ca-c25af6385a11-istio-podinfo" pod "97819a0a-0328-4845-b5ca-c25af6385a11" (UID: "97819a0a-0328-4845-b5ca-c25af6385a11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:27:50.763183 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.763113    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97819a0a-0328-4845-b5ca-c25af6385a11-istio-token" pod "97819a0a-0328-4845-b5ca-c25af6385a11" (UID: "97819a0a-0328-4845-b5ca-c25af6385a11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:50.763298 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.763274    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-istio-envoy" pod "97819a0a-0328-4845-b5ca-c25af6385a11" (UID: "97819a0a-0328-4845-b5ca-c25af6385a11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:27:50.763358 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.763296    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97819a0a-0328-4845-b5ca-c25af6385a11-kube-api-access-xrctp" pod "97819a0a-0328-4845-b5ca-c25af6385a11" (UID: "97819a0a-0328-4845-b5ca-c25af6385a11"). InnerVolumeSpecName "kube-api-access-xrctp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:27:50.862574 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.862480    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97819a0a-0328-4845-b5ca-c25af6385a11-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:50.862574 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.862513    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:50.862574 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.862526    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97819a0a-0328-4845-b5ca-c25af6385a11-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:50.862574 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.862539    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrctp\" (UniqueName: \"kubernetes.io/projected/97819a0a-0328-4845-b5ca-c25af6385a11-kube-api-access-xrctp\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:50.862574 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.862552    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97819a0a-0328-4845-b5ca-c25af6385a11-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:50.862574 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.862565    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97819a0a-0328-4845-b5ca-c25af6385a11-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:27:50.901396 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:50.901345    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:51.043593 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:51.043554    4900 generic.go:356] "Generic (PLEG): container finished" podID="97819a0a-0328-4845-b5ca-c25af6385a11" containerID="5e72147ab15ee0d2590fa42f6238dbbf274746ca75d3a0db8b9d5f07cc4e27d8" exitCode=0
Oct 31 04:27:51.044089 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:51.043597    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"
Oct 31 04:27:51.044089 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:51.043603    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8" event={"ID":"97819a0a-0328-4845-b5ca-c25af6385a11","Type":"ContainerDied","Data":"5e72147ab15ee0d2590fa42f6238dbbf274746ca75d3a0db8b9d5f07cc4e27d8"}
Oct 31 04:27:51.044089 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:51.043683    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8" event={"ID":"97819a0a-0328-4845-b5ca-c25af6385a11","Type":"ContainerDied","Data":"286a2b856fa8c6e81bde0422645ab489e72990462e1d32a9666ff0da6c7ad6c7"}
Oct 31 04:27:51.044089 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:51.043714    4900 scope.go:122] "RemoveContainer" containerID="5e72147ab15ee0d2590fa42f6238dbbf274746ca75d3a0db8b9d5f07cc4e27d8"
Oct 31 04:27:51.053890 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:51.053781    4900 scope.go:122] "RemoveContainer" containerID="5e72147ab15ee0d2590fa42f6238dbbf274746ca75d3a0db8b9d5f07cc4e27d8"
Oct 31 04:27:51.054190 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:27:51.054171    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e72147ab15ee0d2590fa42f6238dbbf274746ca75d3a0db8b9d5f07cc4e27d8\": container with ID starting with 5e72147ab15ee0d2590fa42f6238dbbf274746ca75d3a0db8b9d5f07cc4e27d8 not found: ID does not exist" containerID="5e72147ab15ee0d2590fa42f6238dbbf274746ca75d3a0db8b9d5f07cc4e27d8"
Oct 31 04:27:51.054265 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:51.054200    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e72147ab15ee0d2590fa42f6238dbbf274746ca75d3a0db8b9d5f07cc4e27d8"} err="failed to get container status \"5e72147ab15ee0d2590fa42f6238dbbf274746ca75d3a0db8b9d5f07cc4e27d8\": rpc error: code = NotFound desc = could not find container \"5e72147ab15ee0d2590fa42f6238dbbf274746ca75d3a0db8b9d5f07cc4e27d8\": container with ID starting with 5e72147ab15ee0d2590fa42f6238dbbf274746ca75d3a0db8b9d5f07cc4e27d8 not found: ID does not exist"
Oct 31 04:27:51.151509 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:51.151426    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7" path="/var/lib/kubelet/pods/0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7/volumes"
Oct 31 04:27:51.151979 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:51.151963    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" path="/var/lib/kubelet/pods/82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2/volumes"
Oct 31 04:27:51.152382 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:51.152369    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b7639f-ab0a-4634-9c22-c6408daeb5d0" path="/var/lib/kubelet/pods/c0b7639f-ab0a-4634-9c22-c6408daeb5d0/volumes"
Oct 31 04:27:51.152945 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:51.152926    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca4e29f3-e344-40ea-b9e7-497bef1b2c24" path="/var/lib/kubelet/pods/ca4e29f3-e344-40ea-b9e7-497bef1b2c24/volumes"
Oct 31 04:27:51.208461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:51.208426    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"]
Oct 31 04:27:51.258466 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:51.258426    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2339ed-openshift-default-7c5f9b9459-kq8x8"]
Oct 31 04:27:52.513027 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:52.512991    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:52.867592 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:52.867507    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:53.077774 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:53.077744    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:53.151833 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:53.151737    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97819a0a-0328-4845-b5ca-c25af6385a11" path="/var/lib/kubelet/pods/97819a0a-0328-4845-b5ca-c25af6385a11/volumes"
Oct 31 04:27:53.567183 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:53.567148    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:54.277709 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:54.277673    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:55.303761 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:55.303724    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:55.510962 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:55.510929    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:58.002448 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:58.002412    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:58.836044 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:58.836005    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:59.157502 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.157415    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"]
Oct 31 04:27:59.158253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158229    4900 cpu_manager.go:414] "containerMap: removing container" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerName="main"
Oct 31 04:27:59.158325 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158257    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerName="main"
Oct 31 04:27:59.158325 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158297    4900 cpu_manager.go:414] "containerMap: removing container" podUID="36e17b74-45d2-4d36-842f-1c1868c501e0" containerName="istio-proxy"
Oct 31 04:27:59.158325 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158308    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e17b74-45d2-4d36-842f-1c1868c501e0" containerName="istio-proxy"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158328    4900 cpu_manager.go:414] "containerMap: removing container" podUID="9d97d1f9-e5a1-4768-9940-f81b31df1f44" containerName="istio-proxy"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158336    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d97d1f9-e5a1-4768-9940-f81b31df1f44" containerName="istio-proxy"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158350    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7" containerName="payload-processing"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158357    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7" containerName="payload-processing"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158369    4900 cpu_manager.go:414] "containerMap: removing container" podUID="36e17b74-45d2-4d36-842f-1c1868c501e0" containerName="istio-proxy"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158377    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e17b74-45d2-4d36-842f-1c1868c501e0" containerName="istio-proxy"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158391    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8c101703-159d-4ae6-8e41-e669698c9130" containerName="payload-pre-processing"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158398    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c101703-159d-4ae6-8e41-e669698c9130" containerName="payload-pre-processing"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158414    4900 cpu_manager.go:414] "containerMap: removing container" podUID="48a65552-b7f7-421c-b714-3e3f2f2b3dc2" containerName="main"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158422    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a65552-b7f7-421c-b714-3e3f2f2b3dc2" containerName="main"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158432    4900 cpu_manager.go:414] "containerMap: removing container" podUID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerName="istio-proxy"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158439    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerName="istio-proxy"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158451    4900 cpu_manager.go:414] "containerMap: removing container" podUID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerName="istio-proxy"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158457    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerName="istio-proxy"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158469    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c0b7639f-ab0a-4634-9c22-c6408daeb5d0" containerName="payload-pre-processing"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158476    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b7639f-ab0a-4634-9c22-c6408daeb5d0" containerName="payload-pre-processing"
Oct 31 04:27:59.158481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158487    4900 cpu_manager.go:414] "containerMap: removing container" podUID="f38ad177-3298-48f2-9ec0-18a4a438a664" containerName="revoke-keys"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158494    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38ad177-3298-48f2-9ec0-18a4a438a664" containerName="revoke-keys"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158506    4900 cpu_manager.go:414] "containerMap: removing container" podUID="97819a0a-0328-4845-b5ca-c25af6385a11" containerName="istio-proxy"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158512    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97819a0a-0328-4845-b5ca-c25af6385a11" containerName="istio-proxy"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158614    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="64487e0c-7344-45c3-af5e-23c2681cc64e" containerName="main"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158630    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e17b74-45d2-4d36-842f-1c1868c501e0" containerName="istio-proxy"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158641    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d97d1f9-e5a1-4768-9940-f81b31df1f44" containerName="istio-proxy"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158653    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c361df3-1ec3-42cc-bd4a-b1f9c659bcb7" containerName="payload-processing"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158662    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e17b74-45d2-4d36-842f-1c1868c501e0" containerName="istio-proxy"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158672    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c101703-159d-4ae6-8e41-e669698c9130" containerName="payload-pre-processing"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158681    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a65552-b7f7-421c-b714-3e3f2f2b3dc2" containerName="main"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158689    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerName="istio-proxy"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158698    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="82e8d2a4-3ce9-44f3-892f-d23a0dd55ac2" containerName="istio-proxy"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158707    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b7639f-ab0a-4634-9c22-c6408daeb5d0" containerName="payload-pre-processing"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158717    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38ad177-3298-48f2-9ec0-18a4a438a664" containerName="revoke-keys"
Oct 31 04:27:59.159345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.158727    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="97819a0a-0328-4845-b5ca-c25af6385a11" containerName="istio-proxy"
Oct 31 04:27:59.164065 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.164045    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.215238 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.215203    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"]
Oct 31 04:27:59.246100 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.246056    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:27:59.254405 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.254358    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66691fd9-2ce2-4311-a334-142046c0b848-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.254582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.254552    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.254653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.254618    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.254653 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.254640    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.254742 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.254665    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.254742 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.254686    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66691fd9-2ce2-4311-a334-142046c0b848-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.254874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.254858    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wc8l\" (UniqueName: \"kubernetes.io/projected/66691fd9-2ce2-4311-a334-142046c0b848-kube-api-access-7wc8l\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.254934 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.254918    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.254975 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.254952    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66691fd9-2ce2-4311-a334-142046c0b848-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.263164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.263143    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d39257-openshift-default-dockercfg-bm5z5\""
Oct 31 04:27:59.356401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.356369    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wc8l\" (UniqueName: \"kubernetes.io/projected/66691fd9-2ce2-4311-a334-142046c0b848-kube-api-access-7wc8l\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.356582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.356437    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.356582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.356472    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66691fd9-2ce2-4311-a334-142046c0b848-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.356582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.356544    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66691fd9-2ce2-4311-a334-142046c0b848-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.356750 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.356588    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.356750 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.356622    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.356750 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.356643    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.356750 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.356687    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.356750 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.356715    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66691fd9-2ce2-4311-a334-142046c0b848-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.357097 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.357071    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-istio-data\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.357168 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.357109    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-workload-certs\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.357233 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.357184    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-workload-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.357338 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.357320    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-credential-socket\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.357416 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.357400    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66691fd9-2ce2-4311-a334-142046c0b848-istiod-ca-cert\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.358698 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.358680    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-istio-envoy\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.359055 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.359039    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66691fd9-2ce2-4311-a334-142046c0b848-istio-podinfo\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.419757 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.419689    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66691fd9-2ce2-4311-a334-142046c0b848-istio-token\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.419939 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.419919    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wc8l\" (UniqueName: \"kubernetes.io/projected/66691fd9-2ce2-4311-a334-142046c0b848-kube-api-access-7wc8l\") pod \"e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") " pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.477318 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.477284    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:27:59.891998 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:27:59.891963    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66691fd9_2ce2_4311_a334_142046c0b848.slice/crio-280f98b5370aef1a0d99cf552e379e34907154203cc773d6d1ecbd9c11a7d2b3 WatchSource:0}: Error finding container 280f98b5370aef1a0d99cf552e379e34907154203cc773d6d1ecbd9c11a7d2b3: Status 404 returned error can't find the container with id 280f98b5370aef1a0d99cf552e379e34907154203cc773d6d1ecbd9c11a7d2b3
Oct 31 04:27:59.893667 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.893646    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"]
Oct 31 04:27:59.894288 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.894260    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:27:59.894351 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.894321    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:27:59.894391 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:27:59.894349    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:28:00.093078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:00.093040    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm" event={"ID":"66691fd9-2ce2-4311-a334-142046c0b848","Type":"ContainerStarted","Data":"f3d3e94912cc9a03f12fcba39f2240bfb85d19a1978e4f59c50a6f39fd53a0de"}
Oct 31 04:28:00.093271 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:00.093100    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm" event={"ID":"66691fd9-2ce2-4311-a334-142046c0b848","Type":"ContainerStarted","Data":"280f98b5370aef1a0d99cf552e379e34907154203cc773d6d1ecbd9c11a7d2b3"}
Oct 31 04:28:00.209738 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:00.209639    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm" podStartSLOduration=1.209623937 podStartE2EDuration="1.209623937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:28:00.207992437 +0000 UTC m=+4833.174325781" watchObservedRunningTime="2026-10-31 04:28:00.209623937 +0000 UTC m=+4833.175957281"
Oct 31 04:28:00.477602 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:00.477499    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:28:00.478780 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:00.478757    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 04:28:00.478870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:00.478823    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm" podUID="66691fd9-2ce2-4311-a334-142046c0b848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 04:28:00.857812 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:00.857489    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:01.290552 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:01.290517    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:01.478617 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:01.478578    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 04:28:01.478844 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:01.478636    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm" podUID="66691fd9-2ce2-4311-a334-142046c0b848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 04:28:01.498275 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:01.498244    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:01.922199 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:01.922151    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:02.130005 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:02.129946    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:02.477954 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:02.477925    4900 patch_prober.go:29] interesting pod/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 04:28:02.478438 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:02.477970    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm" podUID="66691fd9-2ce2-4311-a334-142046c0b848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 04:28:02.797295 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:02.797263    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:03.207424 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:03.207344    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"]
Oct 31 04:28:03.207581 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:03.207538    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm" podUID="66691fd9-2ce2-4311-a334-142046c0b848" containerName="istio-proxy" containerID="cri-o://f3d3e94912cc9a03f12fcba39f2240bfb85d19a1978e4f59c50a6f39fd53a0de" gracePeriod=30
Oct 31 04:28:03.275474 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:03.275440    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:05.657369 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:05.657335    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:06.983835 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:28:06.983782    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:28:07.395874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:07.395770    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:09.722966 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:09.722933    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:11.149946 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.149915    4900 generic.go:356] "Generic (PLEG): container finished" podID="66691fd9-2ce2-4311-a334-142046c0b848" containerID="f3d3e94912cc9a03f12fcba39f2240bfb85d19a1978e4f59c50a6f39fd53a0de" exitCode=0
Oct 31 04:28:11.152439 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.152410    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm" event={"ID":"66691fd9-2ce2-4311-a334-142046c0b848","Type":"ContainerDied","Data":"f3d3e94912cc9a03f12fcba39f2240bfb85d19a1978e4f59c50a6f39fd53a0de"}
Oct 31 04:28:11.152549 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.152469    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm" event={"ID":"66691fd9-2ce2-4311-a334-142046c0b848","Type":"ContainerDied","Data":"280f98b5370aef1a0d99cf552e379e34907154203cc773d6d1ecbd9c11a7d2b3"}
Oct 31 04:28:11.152549 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.152495    4900 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="280f98b5370aef1a0d99cf552e379e34907154203cc773d6d1ecbd9c11a7d2b3"
Oct 31 04:28:11.165963 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.165933    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:28:11.285527 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.285498    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-istio-data\") pod \"66691fd9-2ce2-4311-a334-142046c0b848\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") "
Oct 31 04:28:11.285527 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.285539    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-workload-certs\") pod \"66691fd9-2ce2-4311-a334-142046c0b848\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") "
Oct 31 04:28:11.285851 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.285560    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66691fd9-2ce2-4311-a334-142046c0b848-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66691fd9-2ce2-4311-a334-142046c0b848-istio-podinfo\") pod \"66691fd9-2ce2-4311-a334-142046c0b848\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") "
Oct 31 04:28:11.285851 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.285597    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66691fd9-2ce2-4311-a334-142046c0b848-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66691fd9-2ce2-4311-a334-142046c0b848-istiod-ca-cert\") pod \"66691fd9-2ce2-4311-a334-142046c0b848\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") "
Oct 31 04:28:11.285851 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.285618    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66691fd9-2ce2-4311-a334-142046c0b848-istio-token\" (UniqueName: \"kubernetes.io/projected/66691fd9-2ce2-4311-a334-142046c0b848-istio-token\") pod \"66691fd9-2ce2-4311-a334-142046c0b848\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") "
Oct 31 04:28:11.285851 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.285691    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-istio-envoy\") pod \"66691fd9-2ce2-4311-a334-142046c0b848\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") "
Oct 31 04:28:11.285851 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.285723    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-workload-socket\") pod \"66691fd9-2ce2-4311-a334-142046c0b848\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") "
Oct 31 04:28:11.285851 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.285841    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-credential-socket\") pod \"66691fd9-2ce2-4311-a334-142046c0b848\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") "
Oct 31 04:28:11.286171 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.285867    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-istio-data" pod "66691fd9-2ce2-4311-a334-142046c0b848" (UID: "66691fd9-2ce2-4311-a334-142046c0b848"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:11.286171 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.285887    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-workload-certs" pod "66691fd9-2ce2-4311-a334-142046c0b848" (UID: "66691fd9-2ce2-4311-a334-142046c0b848"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:11.286171 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.285961    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66691fd9-2ce2-4311-a334-142046c0b848-kube-api-access-7wc8l\" (UniqueName: \"kubernetes.io/projected/66691fd9-2ce2-4311-a334-142046c0b848-kube-api-access-7wc8l\") pod \"66691fd9-2ce2-4311-a334-142046c0b848\" (UID: \"66691fd9-2ce2-4311-a334-142046c0b848\") "
Oct 31 04:28:11.286171 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.286153    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66691fd9-2ce2-4311-a334-142046c0b848-istiod-ca-cert" pod "66691fd9-2ce2-4311-a334-142046c0b848" (UID: "66691fd9-2ce2-4311-a334-142046c0b848"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:11.286415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.286370    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-workload-socket" pod "66691fd9-2ce2-4311-a334-142046c0b848" (UID: "66691fd9-2ce2-4311-a334-142046c0b848"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:11.286484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.286453    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-credential-socket" pod "66691fd9-2ce2-4311-a334-142046c0b848" (UID: "66691fd9-2ce2-4311-a334-142046c0b848"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:11.286741 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.286719    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:11.286852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.286743    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:11.286852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.286758    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66691fd9-2ce2-4311-a334-142046c0b848-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:11.286852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.286773    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:11.286852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.286812    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:11.288126 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.288099    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-istio-envoy" pod "66691fd9-2ce2-4311-a334-142046c0b848" (UID: "66691fd9-2ce2-4311-a334-142046c0b848"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:11.288126 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.288118    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66691fd9-2ce2-4311-a334-142046c0b848-istio-podinfo" pod "66691fd9-2ce2-4311-a334-142046c0b848" (UID: "66691fd9-2ce2-4311-a334-142046c0b848"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:28:11.288302 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.288157    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66691fd9-2ce2-4311-a334-142046c0b848-kube-api-access-7wc8l" pod "66691fd9-2ce2-4311-a334-142046c0b848" (UID: "66691fd9-2ce2-4311-a334-142046c0b848"). InnerVolumeSpecName "kube-api-access-7wc8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:11.288302 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.288175    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66691fd9-2ce2-4311-a334-142046c0b848-istio-token" pod "66691fd9-2ce2-4311-a334-142046c0b848" (UID: "66691fd9-2ce2-4311-a334-142046c0b848"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:11.387639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.387597    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66691fd9-2ce2-4311-a334-142046c0b848-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:11.387639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.387633    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66691fd9-2ce2-4311-a334-142046c0b848-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:11.387639 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.387642    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66691fd9-2ce2-4311-a334-142046c0b848-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:11.387912 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.387651    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wc8l\" (UniqueName: \"kubernetes.io/projected/66691fd9-2ce2-4311-a334-142046c0b848-kube-api-access-7wc8l\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:11.596177 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:11.596069    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:12.155575 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:12.155543    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"
Oct 31 04:28:12.320435 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:12.320401    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"]
Oct 31 04:28:12.370505 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:12.370474    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d39257-openshift-default-7f9c647847-kp6mm"]
Oct 31 04:28:13.152237 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:13.152202    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66691fd9-2ce2-4311-a334-142046c0b848" path="/var/lib/kubelet/pods/66691fd9-2ce2-4311-a334-142046c0b848/volumes"
Oct 31 04:28:13.423519 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:13.423433    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:13.651777 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:13.651745    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:14.902362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:14.902327    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:15.386550 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:15.386465    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:15.821042 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:15.821006    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:18.531395 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:18.531364    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:20.705006 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:20.704970    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"]
Oct 31 04:28:20.719610 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:20.719572    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"]
Oct 31 04:28:20.754340 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:20.754313    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"]
Oct 31 04:28:20.868998 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:20.868965    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:21.730172 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:21.730142    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:21.987231 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:21.987197    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:22.845644 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:22.845609    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:23.396035 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:23.395994    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:23.626210 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:23.626169    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:23.865076 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:23.865043    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:24.337359 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:24.337322    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:26.877262 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:26.877226    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:28.795091 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:28.795047    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:29.001660 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:29.001630    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:29.209621 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:29.209595    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"]
Oct 31 04:28:29.258524 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:29.258489    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"]
Oct 31 04:28:29.466016 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:29.465934    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:29.799153 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:29.799123    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:30.007310 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:30.007275    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:31.321848 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:31.321779    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:33.906661 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:33.906619    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:34.433173 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:34.433135    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:35.726351 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:35.726308    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" containerName="main" containerID="cri-o://48042c882f7cc6c0615c0cfc1181551412381d7442023a2e8e896d640f17130e" gracePeriod=45
Oct 31 04:28:35.739204 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:35.739174    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main" containerID="cri-o://be3400c115b8296d0004971f09224dcb890b1d20dda4d158c94ffad189ea3460" gracePeriod=45
Oct 31 04:28:35.773478 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:35.773447    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main" containerID="cri-o://b10e6a986d3cd85f095dfe6a70da7ed3849a7702550421a96dcc78457774fac2" gracePeriod=45
Oct 31 04:28:35.779112 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:35.779077    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": dial tcp 10.133.0.50:8000: connect: connection refused"
Oct 31 04:28:36.017590 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.017566    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 04:28:36.033302 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.033278    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 04:28:36.049612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.049575    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-tmp-dir\") pod \"0962c273-9b23-4c27-9e40-127a53cd190a\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") "
Oct 31 04:28:36.049774 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.049626    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-dshm\") pod \"0962c273-9b23-4c27-9e40-127a53cd190a\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") "
Oct 31 04:28:36.049774 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.049746    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0962c273-9b23-4c27-9e40-127a53cd190a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0962c273-9b23-4c27-9e40-127a53cd190a-tls-certs\") pod \"0962c273-9b23-4c27-9e40-127a53cd190a\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") "
Oct 31 04:28:36.049933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.049832    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-home\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-home\") pod \"0962c273-9b23-4c27-9e40-127a53cd190a\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") "
Oct 31 04:28:36.049933 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.049891    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-model-cache\") pod \"0962c273-9b23-4c27-9e40-127a53cd190a\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") "
Oct 31 04:28:36.050134 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.049976    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0962c273-9b23-4c27-9e40-127a53cd190a-kube-api-access-zmd26\" (UniqueName: \"kubernetes.io/projected/0962c273-9b23-4c27-9e40-127a53cd190a-kube-api-access-zmd26\") pod \"0962c273-9b23-4c27-9e40-127a53cd190a\" (UID: \"0962c273-9b23-4c27-9e40-127a53cd190a\") "
Oct 31 04:28:36.050134 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.049886    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-tmp-dir" pod "0962c273-9b23-4c27-9e40-127a53cd190a" (UID: "0962c273-9b23-4c27-9e40-127a53cd190a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.050134 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.050088    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-home" pod "0962c273-9b23-4c27-9e40-127a53cd190a" (UID: "0962c273-9b23-4c27-9e40-127a53cd190a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.050316 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.050217    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-model-cache" pod "0962c273-9b23-4c27-9e40-127a53cd190a" (UID: "0962c273-9b23-4c27-9e40-127a53cd190a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.050378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.050353    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.050378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.050366    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.050378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.050375    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.052313 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.052286    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0962c273-9b23-4c27-9e40-127a53cd190a-kube-api-access-zmd26" pod "0962c273-9b23-4c27-9e40-127a53cd190a" (UID: "0962c273-9b23-4c27-9e40-127a53cd190a"). InnerVolumeSpecName "kube-api-access-zmd26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:36.052423 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.052346    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-dshm" pod "0962c273-9b23-4c27-9e40-127a53cd190a" (UID: "0962c273-9b23-4c27-9e40-127a53cd190a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.052423 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.052376    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0962c273-9b23-4c27-9e40-127a53cd190a-tls-certs" pod "0962c273-9b23-4c27-9e40-127a53cd190a" (UID: "0962c273-9b23-4c27-9e40-127a53cd190a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:36.059348 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.059328    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 04:28:36.151084 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151055    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-model-cache\") pod \"8e0f8b95-9b82-406c-8909-f8a11c81a753\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") "
Oct 31 04:28:36.151279 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151105    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/397fbdbf-8683-4d96-aa94-5e8ea77a785d-kube-api-access-s8s5d\" (UniqueName: \"kubernetes.io/projected/397fbdbf-8683-4d96-aa94-5e8ea77a785d-kube-api-access-s8s5d\") pod \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") "
Oct 31 04:28:36.151279 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151134    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-tmp-dir\") pod \"8e0f8b95-9b82-406c-8909-f8a11c81a753\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") "
Oct 31 04:28:36.151279 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151156    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-home\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-home\") pod \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") "
Oct 31 04:28:36.151279 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151188    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-tmp-dir\") pod \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") "
Oct 31 04:28:36.151279 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151222    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e0f8b95-9b82-406c-8909-f8a11c81a753-kube-api-access-r6h4n\" (UniqueName: \"kubernetes.io/projected/8e0f8b95-9b82-406c-8909-f8a11c81a753-kube-api-access-r6h4n\") pod \"8e0f8b95-9b82-406c-8909-f8a11c81a753\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") "
Oct 31 04:28:36.151279 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151252    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/397fbdbf-8683-4d96-aa94-5e8ea77a785d-tls-certs\" (UniqueName: \"kubernetes.io/secret/397fbdbf-8683-4d96-aa94-5e8ea77a785d-tls-certs\") pod \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") "
Oct 31 04:28:36.151612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151290    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-home\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-home\") pod \"8e0f8b95-9b82-406c-8909-f8a11c81a753\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") "
Oct 31 04:28:36.151612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151330    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-dshm\") pod \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") "
Oct 31 04:28:36.151612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151349    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-model-cache" pod "8e0f8b95-9b82-406c-8909-f8a11c81a753" (UID: "8e0f8b95-9b82-406c-8909-f8a11c81a753"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.151612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151404    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-tmp-dir" pod "8e0f8b95-9b82-406c-8909-f8a11c81a753" (UID: "8e0f8b95-9b82-406c-8909-f8a11c81a753"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.151612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151435    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-dshm\") pod \"8e0f8b95-9b82-406c-8909-f8a11c81a753\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") "
Oct 31 04:28:36.151612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151469    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e0f8b95-9b82-406c-8909-f8a11c81a753-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0f8b95-9b82-406c-8909-f8a11c81a753-tls-certs\") pod \"8e0f8b95-9b82-406c-8909-f8a11c81a753\" (UID: \"8e0f8b95-9b82-406c-8909-f8a11c81a753\") "
Oct 31 04:28:36.151612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151560    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-model-cache\") pod \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\" (UID: \"397fbdbf-8683-4d96-aa94-5e8ea77a785d\") "
Oct 31 04:28:36.152036 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151651    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-home" pod "8e0f8b95-9b82-406c-8909-f8a11c81a753" (UID: "8e0f8b95-9b82-406c-8909-f8a11c81a753"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.152036 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.151812    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-home" pod "397fbdbf-8683-4d96-aa94-5e8ea77a785d" (UID: "397fbdbf-8683-4d96-aa94-5e8ea77a785d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.152160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.152034    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-model-cache" pod "397fbdbf-8683-4d96-aa94-5e8ea77a785d" (UID: "397fbdbf-8683-4d96-aa94-5e8ea77a785d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.152160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.152044    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0962c273-9b23-4c27-9e40-127a53cd190a-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.152160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.152065    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.152160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.152081    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.152160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.152094    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.152160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.152106    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.152160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.152120    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0962c273-9b23-4c27-9e40-127a53cd190a-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.152160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.152133    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmd26\" (UniqueName: \"kubernetes.io/projected/0962c273-9b23-4c27-9e40-127a53cd190a-kube-api-access-zmd26\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.152603 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.152334    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-tmp-dir" pod "397fbdbf-8683-4d96-aa94-5e8ea77a785d" (UID: "397fbdbf-8683-4d96-aa94-5e8ea77a785d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.154266 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.154240    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/397fbdbf-8683-4d96-aa94-5e8ea77a785d-tls-certs" pod "397fbdbf-8683-4d96-aa94-5e8ea77a785d" (UID: "397fbdbf-8683-4d96-aa94-5e8ea77a785d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:36.154405 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.154357    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e0f8b95-9b82-406c-8909-f8a11c81a753-kube-api-access-r6h4n" pod "8e0f8b95-9b82-406c-8909-f8a11c81a753" (UID: "8e0f8b95-9b82-406c-8909-f8a11c81a753"). InnerVolumeSpecName "kube-api-access-r6h4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:36.154475 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.154440    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-dshm" pod "397fbdbf-8683-4d96-aa94-5e8ea77a785d" (UID: "397fbdbf-8683-4d96-aa94-5e8ea77a785d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.155136 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.155104    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e0f8b95-9b82-406c-8909-f8a11c81a753-tls-certs" pod "8e0f8b95-9b82-406c-8909-f8a11c81a753" (UID: "8e0f8b95-9b82-406c-8909-f8a11c81a753"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:36.155305 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.155283    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-dshm" pod "8e0f8b95-9b82-406c-8909-f8a11c81a753" (UID: "8e0f8b95-9b82-406c-8909-f8a11c81a753"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:36.156368 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.156343    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397fbdbf-8683-4d96-aa94-5e8ea77a785d-kube-api-access-s8s5d" pod "397fbdbf-8683-4d96-aa94-5e8ea77a785d" (UID: "397fbdbf-8683-4d96-aa94-5e8ea77a785d"). InnerVolumeSpecName "kube-api-access-s8s5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:36.253005 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.252922    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.253005 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.252954    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6h4n\" (UniqueName: \"kubernetes.io/projected/8e0f8b95-9b82-406c-8909-f8a11c81a753-kube-api-access-r6h4n\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.253005 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.252967    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/397fbdbf-8683-4d96-aa94-5e8ea77a785d-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.253005 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.252981    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.253005 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.252993    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e0f8b95-9b82-406c-8909-f8a11c81a753-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.253005 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.253004    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e0f8b95-9b82-406c-8909-f8a11c81a753-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.253485 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.253016    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397fbdbf-8683-4d96-aa94-5e8ea77a785d-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.253485 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.253028    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8s5d\" (UniqueName: \"kubernetes.io/projected/397fbdbf-8683-4d96-aa94-5e8ea77a785d-kube-api-access-s8s5d\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:36.288592 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.288555    4900 generic.go:356] "Generic (PLEG): container finished" podID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerID="b10e6a986d3cd85f095dfe6a70da7ed3849a7702550421a96dcc78457774fac2" exitCode=0
Oct 31 04:28:36.288592 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.288588    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"
Oct 31 04:28:36.288853 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.288630    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" event={"ID":"8e0f8b95-9b82-406c-8909-f8a11c81a753","Type":"ContainerDied","Data":"b10e6a986d3cd85f095dfe6a70da7ed3849a7702550421a96dcc78457774fac2"}
Oct 31 04:28:36.288853 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.288682    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl" event={"ID":"8e0f8b95-9b82-406c-8909-f8a11c81a753","Type":"ContainerDied","Data":"37a58c6fdd18769c9d33a1e8110981ebfc5b2fc0e9f2da2840f86b9eedf69b9a"}
Oct 31 04:28:36.288853 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.288708    4900 scope.go:122] "RemoveContainer" containerID="b10e6a986d3cd85f095dfe6a70da7ed3849a7702550421a96dcc78457774fac2"
Oct 31 04:28:36.290135 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.290113    4900 generic.go:356] "Generic (PLEG): container finished" podID="0962c273-9b23-4c27-9e40-127a53cd190a" containerID="48042c882f7cc6c0615c0cfc1181551412381d7442023a2e8e896d640f17130e" exitCode=0
Oct 31 04:28:36.290246 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.290156    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"
Oct 31 04:28:36.290246 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.290217    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" event={"ID":"0962c273-9b23-4c27-9e40-127a53cd190a","Type":"ContainerDied","Data":"48042c882f7cc6c0615c0cfc1181551412381d7442023a2e8e896d640f17130e"}
Oct 31 04:28:36.290357 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.290254    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th" event={"ID":"0962c273-9b23-4c27-9e40-127a53cd190a","Type":"ContainerDied","Data":"e0e57c5630279140a5c8ba9b8a8e9b1c070ccdebaa0d603518e756858ace3c4c"}
Oct 31 04:28:36.291692 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.291669    4900 generic.go:356] "Generic (PLEG): container finished" podID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerID="be3400c115b8296d0004971f09224dcb890b1d20dda4d158c94ffad189ea3460" exitCode=0
Oct 31 04:28:36.291849 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.291713    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"
Oct 31 04:28:36.291849 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.291716    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" event={"ID":"397fbdbf-8683-4d96-aa94-5e8ea77a785d","Type":"ContainerDied","Data":"be3400c115b8296d0004971f09224dcb890b1d20dda4d158c94ffad189ea3460"}
Oct 31 04:28:36.291849 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.291820    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th" event={"ID":"397fbdbf-8683-4d96-aa94-5e8ea77a785d","Type":"ContainerDied","Data":"c560be3d64d8423b955de775f952222f9e22af667eaa86dfd9c9a3d499f41dcd"}
Oct 31 04:28:36.299544 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.299527    4900 scope.go:122] "RemoveContainer" containerID="b10e6a986d3cd85f095dfe6a70da7ed3849a7702550421a96dcc78457774fac2"
Oct 31 04:28:36.299960 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:28:36.299843    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10e6a986d3cd85f095dfe6a70da7ed3849a7702550421a96dcc78457774fac2\": container with ID starting with b10e6a986d3cd85f095dfe6a70da7ed3849a7702550421a96dcc78457774fac2 not found: ID does not exist" containerID="b10e6a986d3cd85f095dfe6a70da7ed3849a7702550421a96dcc78457774fac2"
Oct 31 04:28:36.299960 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.299876    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10e6a986d3cd85f095dfe6a70da7ed3849a7702550421a96dcc78457774fac2"} err="failed to get container status \"b10e6a986d3cd85f095dfe6a70da7ed3849a7702550421a96dcc78457774fac2\": rpc error: code = NotFound desc = could not find container \"b10e6a986d3cd85f095dfe6a70da7ed3849a7702550421a96dcc78457774fac2\": container with ID starting with b10e6a986d3cd85f095dfe6a70da7ed3849a7702550421a96dcc78457774fac2 not found: ID does not exist"
Oct 31 04:28:36.299960 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.299899    4900 scope.go:122] "RemoveContainer" containerID="48042c882f7cc6c0615c0cfc1181551412381d7442023a2e8e896d640f17130e"
Oct 31 04:28:36.309737 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.309720    4900 scope.go:122] "RemoveContainer" containerID="48042c882f7cc6c0615c0cfc1181551412381d7442023a2e8e896d640f17130e"
Oct 31 04:28:36.310039 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:28:36.310017    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48042c882f7cc6c0615c0cfc1181551412381d7442023a2e8e896d640f17130e\": container with ID starting with 48042c882f7cc6c0615c0cfc1181551412381d7442023a2e8e896d640f17130e not found: ID does not exist" containerID="48042c882f7cc6c0615c0cfc1181551412381d7442023a2e8e896d640f17130e"
Oct 31 04:28:36.310132 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.310042    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48042c882f7cc6c0615c0cfc1181551412381d7442023a2e8e896d640f17130e"} err="failed to get container status \"48042c882f7cc6c0615c0cfc1181551412381d7442023a2e8e896d640f17130e\": rpc error: code = NotFound desc = could not find container \"48042c882f7cc6c0615c0cfc1181551412381d7442023a2e8e896d640f17130e\": container with ID starting with 48042c882f7cc6c0615c0cfc1181551412381d7442023a2e8e896d640f17130e not found: ID does not exist"
Oct 31 04:28:36.310132 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.310058    4900 scope.go:122] "RemoveContainer" containerID="be3400c115b8296d0004971f09224dcb890b1d20dda4d158c94ffad189ea3460"
Oct 31 04:28:36.317740 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.317723    4900 scope.go:122] "RemoveContainer" containerID="be3400c115b8296d0004971f09224dcb890b1d20dda4d158c94ffad189ea3460"
Oct 31 04:28:36.317999 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:28:36.317980    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3400c115b8296d0004971f09224dcb890b1d20dda4d158c94ffad189ea3460\": container with ID starting with be3400c115b8296d0004971f09224dcb890b1d20dda4d158c94ffad189ea3460 not found: ID does not exist" containerID="be3400c115b8296d0004971f09224dcb890b1d20dda4d158c94ffad189ea3460"
Oct 31 04:28:36.318059 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.318005    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3400c115b8296d0004971f09224dcb890b1d20dda4d158c94ffad189ea3460"} err="failed to get container status \"be3400c115b8296d0004971f09224dcb890b1d20dda4d158c94ffad189ea3460\": rpc error: code = NotFound desc = could not find container \"be3400c115b8296d0004971f09224dcb890b1d20dda4d158c94ffad189ea3460\": container with ID starting with be3400c115b8296d0004971f09224dcb890b1d20dda4d158c94ffad189ea3460 not found: ID does not exist"
Oct 31 04:28:36.473887 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.473853    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"]
Oct 31 04:28:36.479197 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.479174    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/distinct-2-w2-b59563-kserve-b97b5fb76-4v2th"]
Oct 31 04:28:36.587237 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.587153    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"]
Oct 31 04:28:36.589903 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.589882    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/premium-w2-b59563-kserve-74856ddf84-hh7th"]
Oct 31 04:28:36.705642 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.705604    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"]
Oct 31 04:28:36.723965 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.723936    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b59563/simulator-w2-b59563-kserve-c95f846fc-jhgcl"]
Oct 31 04:28:36.871375 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:36.871264    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:36.985340 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:28:36.985299    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:28:37.095683 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:37.095652    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:37.152475 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:37.152400    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" path="/var/lib/kubelet/pods/0962c273-9b23-4c27-9e40-127a53cd190a/volumes"
Oct 31 04:28:37.152850 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:37.152834    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" path="/var/lib/kubelet/pods/397fbdbf-8683-4d96-aa94-5e8ea77a785d/volumes"
Oct 31 04:28:37.153213 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:37.153201    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" path="/var/lib/kubelet/pods/8e0f8b95-9b82-406c-8909-f8a11c81a753/volumes"
Oct 31 04:28:38.043924 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:38.043877    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:38.978651 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:38.978619    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:41.281886 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:41.281854    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:43.191906 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.191875    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"]
Oct 31 04:28:43.192289 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.192071    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5" podUID="9f16f01c-772c-4277-8519-8b017eceac4b" containerName="maas-api" containerID="cri-o://72da6a4818b4d09a5b8ea85b37ed50d66d395aee9da1b3f589424175f0c42e4a" gracePeriod=30
Oct 31 04:28:43.237727 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:28:43.237688    4900 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f16f01c_772c_4277_8519_8b017eceac4b.slice/crio-72da6a4818b4d09a5b8ea85b37ed50d66d395aee9da1b3f589424175f0c42e4a.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 04:28:43.331260 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.331227    4900 generic.go:356] "Generic (PLEG): container finished" podID="9f16f01c-772c-4277-8519-8b017eceac4b" containerID="72da6a4818b4d09a5b8ea85b37ed50d66d395aee9da1b3f589424175f0c42e4a" exitCode=0
Oct 31 04:28:43.331444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.331300    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5" event={"ID":"9f16f01c-772c-4277-8519-8b017eceac4b","Type":"ContainerDied","Data":"72da6a4818b4d09a5b8ea85b37ed50d66d395aee9da1b3f589424175f0c42e4a"}
Oct 31 04:28:43.365351 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.365318    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:43.441436 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.441406    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 04:28:43.522864 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.522831    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f16f01c-772c-4277-8519-8b017eceac4b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f16f01c-772c-4277-8519-8b017eceac4b-openshift-service-ca\") pod \"9f16f01c-772c-4277-8519-8b017eceac4b\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") "
Oct 31 04:28:43.523037 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.522873    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-metrics-certs\") pod \"9f16f01c-772c-4277-8519-8b017eceac4b\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") "
Oct 31 04:28:43.523037 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.522907    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f16f01c-772c-4277-8519-8b017eceac4b-kube-api-access-j4v4n\" (UniqueName: \"kubernetes.io/projected/9f16f01c-772c-4277-8519-8b017eceac4b-kube-api-access-j4v4n\") pod \"9f16f01c-772c-4277-8519-8b017eceac4b\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") "
Oct 31 04:28:43.523037 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.522962    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-maas-api-tls\") pod \"9f16f01c-772c-4277-8519-8b017eceac4b\" (UID: \"9f16f01c-772c-4277-8519-8b017eceac4b\") "
Oct 31 04:28:43.523411 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.523376    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f16f01c-772c-4277-8519-8b017eceac4b-openshift-service-ca" pod "9f16f01c-772c-4277-8519-8b017eceac4b" (UID: "9f16f01c-772c-4277-8519-8b017eceac4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:28:43.525119 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.525088    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-metrics-certs" pod "9f16f01c-772c-4277-8519-8b017eceac4b" (UID: "9f16f01c-772c-4277-8519-8b017eceac4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:43.525119 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.525110    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-maas-api-tls" pod "9f16f01c-772c-4277-8519-8b017eceac4b" (UID: "9f16f01c-772c-4277-8519-8b017eceac4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:43.525281 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.525090    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f16f01c-772c-4277-8519-8b017eceac4b-kube-api-access-j4v4n" pod "9f16f01c-772c-4277-8519-8b017eceac4b" (UID: "9f16f01c-772c-4277-8519-8b017eceac4b"). InnerVolumeSpecName "kube-api-access-j4v4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:43.624590 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.624552    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:43.624590 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.624582    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f16f01c-772c-4277-8519-8b017eceac4b-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:43.624590 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.624591    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f16f01c-772c-4277-8519-8b017eceac4b-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:43.624879 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.624604    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4v4n\" (UniqueName: \"kubernetes.io/projected/9f16f01c-772c-4277-8519-8b017eceac4b-kube-api-access-j4v4n\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:43.817354 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:43.817276    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:44.230209 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.230098    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main" containerID="cri-o://c18a3bb47e343cb6ef7024cf0be91a36c8b4f7dcc794624bf42c4ae546f5718a" gracePeriod=45
Oct 31 04:28:44.277782 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.277746    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" containerName="main" containerID="cri-o://7a66eaa92636a09d8de4d93d48f6844816dfd4a3c9e42273083ac3c4484b42ea" gracePeriod=45
Oct 31 04:28:44.339313 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.339286    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"
Oct 31 04:28:44.339432 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.339304    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5" event={"ID":"9f16f01c-772c-4277-8519-8b017eceac4b","Type":"ContainerDied","Data":"5d492f579d445e7de1598956065616ddb725d97d34df93c56a7401f89cec1362"}
Oct 31 04:28:44.339432 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.339370    4900 scope.go:122] "RemoveContainer" containerID="72da6a4818b4d09a5b8ea85b37ed50d66d395aee9da1b3f589424175f0c42e4a"
Oct 31 04:28:44.503771 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.503743    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"]
Oct 31 04:28:44.520636 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.520611    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 04:28:44.548224 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.548203    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 04:28:44.552962 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.552945    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-b59563-ffd6fc897-q7wb5"]
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.634225    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-home\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-home\") pod \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") "
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.634271    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0371044e-9bea-4b59-8329-2aa223d2c126-tls-certs\" (UniqueName: \"kubernetes.io/secret/0371044e-9bea-4b59-8329-2aa223d2c126-tls-certs\") pod \"0371044e-9bea-4b59-8329-2aa223d2c126\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") "
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.634319    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-kube-api-access-bm579\" (UniqueName: \"kubernetes.io/projected/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-kube-api-access-bm579\") pod \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") "
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.634646    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-home" pod "6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" (UID: "6c308b1a-70f2-45b0-9a25-d2f8501a3b1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.634716    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-tls-certs\") pod \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") "
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.634758    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0371044e-9bea-4b59-8329-2aa223d2c126-kube-api-access-lc8ng\" (UniqueName: \"kubernetes.io/projected/0371044e-9bea-4b59-8329-2aa223d2c126-kube-api-access-lc8ng\") pod \"0371044e-9bea-4b59-8329-2aa223d2c126\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") "
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.634848    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-model-cache\") pod \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") "
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.634879    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-dshm\") pod \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") "
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.634916    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-home\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-home\") pod \"0371044e-9bea-4b59-8329-2aa223d2c126\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") "
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.634943    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-tmp-dir\") pod \"0371044e-9bea-4b59-8329-2aa223d2c126\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") "
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.634974    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-dshm\") pod \"0371044e-9bea-4b59-8329-2aa223d2c126\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") "
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.635069    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-tmp-dir\") pod \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\" (UID: \"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c\") "
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.635094    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-model-cache\") pod \"0371044e-9bea-4b59-8329-2aa223d2c126\" (UID: \"0371044e-9bea-4b59-8329-2aa223d2c126\") "
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.635463    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.636212    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-model-cache" pod "6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" (UID: "6c308b1a-70f2-45b0-9a25-d2f8501a3b1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.636755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.636502    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-home" pod "0371044e-9bea-4b59-8329-2aa223d2c126" (UID: "0371044e-9bea-4b59-8329-2aa223d2c126"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.637755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.637194    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0371044e-9bea-4b59-8329-2aa223d2c126-tls-certs" pod "0371044e-9bea-4b59-8329-2aa223d2c126" (UID: "0371044e-9bea-4b59-8329-2aa223d2c126"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:44.637755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.637225    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-tmp-dir" pod "6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" (UID: "6c308b1a-70f2-45b0-9a25-d2f8501a3b1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.637755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.637235    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-tmp-dir" pod "0371044e-9bea-4b59-8329-2aa223d2c126" (UID: "0371044e-9bea-4b59-8329-2aa223d2c126"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.637755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.637314    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-tls-certs" pod "6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" (UID: "6c308b1a-70f2-45b0-9a25-d2f8501a3b1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:28:44.637755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.637608    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-kube-api-access-bm579" pod "6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" (UID: "6c308b1a-70f2-45b0-9a25-d2f8501a3b1c"). InnerVolumeSpecName "kube-api-access-bm579". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:44.638022 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.637863    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-model-cache" pod "0371044e-9bea-4b59-8329-2aa223d2c126" (UID: "0371044e-9bea-4b59-8329-2aa223d2c126"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.638398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.638364    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-dshm" pod "6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" (UID: "6c308b1a-70f2-45b0-9a25-d2f8501a3b1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.640394 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.640366    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0371044e-9bea-4b59-8329-2aa223d2c126-kube-api-access-lc8ng" pod "0371044e-9bea-4b59-8329-2aa223d2c126" (UID: "0371044e-9bea-4b59-8329-2aa223d2c126"). InnerVolumeSpecName "kube-api-access-lc8ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:28:44.640901 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.640881    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-dshm" pod "0371044e-9bea-4b59-8329-2aa223d2c126" (UID: "0371044e-9bea-4b59-8329-2aa223d2c126"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:28:44.736507 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.736472    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.736507 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.736503    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.736507 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.736513    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0371044e-9bea-4b59-8329-2aa223d2c126-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.736772 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.736523    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm579\" (UniqueName: \"kubernetes.io/projected/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-kube-api-access-bm579\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.736772 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.736533    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.736772 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.736542    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc8ng\" (UniqueName: \"kubernetes.io/projected/0371044e-9bea-4b59-8329-2aa223d2c126-kube-api-access-lc8ng\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.736772 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.736550    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.736772 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.736559    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.736772 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.736567    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.736772 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.736574    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:44.736772 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:44.736581    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0371044e-9bea-4b59-8329-2aa223d2c126-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:28:45.152493 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.152463    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f16f01c-772c-4277-8519-8b017eceac4b" path="/var/lib/kubelet/pods/9f16f01c-772c-4277-8519-8b017eceac4b/volumes"
Oct 31 04:28:45.346503 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.346465    4900 generic.go:356] "Generic (PLEG): container finished" podID="0371044e-9bea-4b59-8329-2aa223d2c126" containerID="7a66eaa92636a09d8de4d93d48f6844816dfd4a3c9e42273083ac3c4484b42ea" exitCode=0
Oct 31 04:28:45.346503 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.346507    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"
Oct 31 04:28:45.347101 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.346555    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" event={"ID":"0371044e-9bea-4b59-8329-2aa223d2c126","Type":"ContainerDied","Data":"7a66eaa92636a09d8de4d93d48f6844816dfd4a3c9e42273083ac3c4484b42ea"}
Oct 31 04:28:45.347101 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.346599    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz" event={"ID":"0371044e-9bea-4b59-8329-2aa223d2c126","Type":"ContainerDied","Data":"3c6cf126bb1dc8fb623e19687544df5aedf91ee47c774c62455ac56da25fea18"}
Oct 31 04:28:45.347101 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.346621    4900 scope.go:122] "RemoveContainer" containerID="7a66eaa92636a09d8de4d93d48f6844816dfd4a3c9e42273083ac3c4484b42ea"
Oct 31 04:28:45.348035 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.348016    4900 generic.go:356] "Generic (PLEG): container finished" podID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerID="c18a3bb47e343cb6ef7024cf0be91a36c8b4f7dcc794624bf42c4ae546f5718a" exitCode=0
Oct 31 04:28:45.348134 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.348044    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" event={"ID":"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c","Type":"ContainerDied","Data":"c18a3bb47e343cb6ef7024cf0be91a36c8b4f7dcc794624bf42c4ae546f5718a"}
Oct 31 04:28:45.348134 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.348058    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"
Oct 31 04:28:45.348134 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.348084    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj" event={"ID":"6c308b1a-70f2-45b0-9a25-d2f8501a3b1c","Type":"ContainerDied","Data":"5d3e5474c472f27b3f72070e76e0d71dacafb31c6052520daf5a9bf7e4f646eb"}
Oct 31 04:28:45.355106 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.355090    4900 scope.go:122] "RemoveContainer" containerID="7a66eaa92636a09d8de4d93d48f6844816dfd4a3c9e42273083ac3c4484b42ea"
Oct 31 04:28:45.355338 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:28:45.355320    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a66eaa92636a09d8de4d93d48f6844816dfd4a3c9e42273083ac3c4484b42ea\": container with ID starting with 7a66eaa92636a09d8de4d93d48f6844816dfd4a3c9e42273083ac3c4484b42ea not found: ID does not exist" containerID="7a66eaa92636a09d8de4d93d48f6844816dfd4a3c9e42273083ac3c4484b42ea"
Oct 31 04:28:45.355445 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.355345    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a66eaa92636a09d8de4d93d48f6844816dfd4a3c9e42273083ac3c4484b42ea"} err="failed to get container status \"7a66eaa92636a09d8de4d93d48f6844816dfd4a3c9e42273083ac3c4484b42ea\": rpc error: code = NotFound desc = could not find container \"7a66eaa92636a09d8de4d93d48f6844816dfd4a3c9e42273083ac3c4484b42ea\": container with ID starting with 7a66eaa92636a09d8de4d93d48f6844816dfd4a3c9e42273083ac3c4484b42ea not found: ID does not exist"
Oct 31 04:28:45.355445 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.355360    4900 scope.go:122] "RemoveContainer" containerID="c18a3bb47e343cb6ef7024cf0be91a36c8b4f7dcc794624bf42c4ae546f5718a"
Oct 31 04:28:45.362772 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.362757    4900 scope.go:122] "RemoveContainer" containerID="c18a3bb47e343cb6ef7024cf0be91a36c8b4f7dcc794624bf42c4ae546f5718a"
Oct 31 04:28:45.363078 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:28:45.363061    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c18a3bb47e343cb6ef7024cf0be91a36c8b4f7dcc794624bf42c4ae546f5718a\": container with ID starting with c18a3bb47e343cb6ef7024cf0be91a36c8b4f7dcc794624bf42c4ae546f5718a not found: ID does not exist" containerID="c18a3bb47e343cb6ef7024cf0be91a36c8b4f7dcc794624bf42c4ae546f5718a"
Oct 31 04:28:45.363120 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.363085    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c18a3bb47e343cb6ef7024cf0be91a36c8b4f7dcc794624bf42c4ae546f5718a"} err="failed to get container status \"c18a3bb47e343cb6ef7024cf0be91a36c8b4f7dcc794624bf42c4ae546f5718a\": rpc error: code = NotFound desc = could not find container \"c18a3bb47e343cb6ef7024cf0be91a36c8b4f7dcc794624bf42c4ae546f5718a\": container with ID starting with c18a3bb47e343cb6ef7024cf0be91a36c8b4f7dcc794624bf42c4ae546f5718a not found: ID does not exist"
Oct 31 04:28:45.509899 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.509862    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"]
Oct 31 04:28:45.559612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.559577    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/distinct-2-w4-9360eb-kserve-6c7698c44c-8gjrj"]
Oct 31 04:28:45.667684 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.667649    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"]
Oct 31 04:28:45.673818 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:45.673778    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9360eb/simulator-w4-9360eb-kserve-7449fb8744-ff5cz"]
Oct 31 04:28:46.954132 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:46.954101    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:47.151403 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:47.151368    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" path="/var/lib/kubelet/pods/0371044e-9bea-4b59-8329-2aa223d2c126/volumes"
Oct 31 04:28:47.151914 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:47.151893    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" path="/var/lib/kubelet/pods/6c308b1a-70f2-45b0-9a25-d2f8501a3b1c/volumes"
Oct 31 04:28:47.197307 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:47.197260    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:49.483390 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:49.483336    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:49.691673 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:49.691638    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:54.997353 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:54.997322    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:55.204300 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:55.204271    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:57.315019 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:57.314982    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:57.538822 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:57.538772    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:28:58.553939 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:28:58.553901    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:01.599836 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:01.599805    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:02.073950 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:02.073917    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:04.139614 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:04.139581    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:04.348323 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:04.348289    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:05.846350 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:05.846320    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:06.083906 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:06.083871    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:06.983001 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:29:06.982969    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:29:07.183732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:07.183682    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:09.689251 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:09.689194    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:10.061261 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:10.061228    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:10.258291 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:10.258255    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:12.774549 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:12.774514    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:15.315798 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:15.315766    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5cb476b87b-tk2rk_b7b8d483-9270-45ec-94dc-d3e1bc625ca2/manager/0.log"
Oct 31 04:29:19.300852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:19.300783    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:19.519940 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:19.519906    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:22.373869 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:22.373825    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:22.980406 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:22.980365    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:23.243923 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:23.243886    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:25.895734 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:25.895702    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:26.122888 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:26.122853    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:26.360262 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:26.360224    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:26.683090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:26.682997    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:27.119356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:27.119308    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:27.360038 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:27.359997    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:29.220657 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:29.220625    4900 scope.go:122] "RemoveContainer" containerID="e324d98e48fdda0bd2f0cb6d93148656895421d052cb1c277363c14b10e54afe"
Oct 31 04:29:29.229145 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:29.229129    4900 scope.go:122] "RemoveContainer" containerID="63a4ea044c1577f38249a267b2727dac847f181218460521cc2b18486c1ebb3d"
Oct 31 04:29:29.633000 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:29.632961    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:29.861175 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:29.861141    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:32.846295 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:32.846262    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:33.068338 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:33.068308    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:36.983977 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:29:36.983938    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:29:37.877741 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:37.877704    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:42.823048 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:42.823013    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:47.119635 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:47.119586    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:47.393676 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:47.393591    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:47.717177 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:47.717086    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:49.367206 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:49.367174    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:52.965238 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:52.965189    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:29:58.259763 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:29:58.259732    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:00.158303 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158266    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"]
Oct 31 04:30:00.158694 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158631    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" containerName="main"
Oct 31 04:30:00.158694 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158642    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" containerName="main"
Oct 31 04:30:00.158694 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158659    4900 cpu_manager.go:414] "containerMap: removing container" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main"
Oct 31 04:30:00.158694 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158664    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main"
Oct 31 04:30:00.158694 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158671    4900 cpu_manager.go:414] "containerMap: removing container" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main"
Oct 31 04:30:00.158694 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158676    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main"
Oct 31 04:30:00.158694 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158691    4900 cpu_manager.go:414] "containerMap: removing container" podUID="66691fd9-2ce2-4311-a334-142046c0b848" containerName="istio-proxy"
Oct 31 04:30:00.158694 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158697    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66691fd9-2ce2-4311-a334-142046c0b848" containerName="istio-proxy"
Oct 31 04:30:00.158980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158706    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main"
Oct 31 04:30:00.158980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158710    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main"
Oct 31 04:30:00.158980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158722    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" containerName="main"
Oct 31 04:30:00.158980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158727    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" containerName="main"
Oct 31 04:30:00.158980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158741    4900 cpu_manager.go:414] "containerMap: removing container" podUID="9f16f01c-772c-4277-8519-8b017eceac4b" containerName="maas-api"
Oct 31 04:30:00.158980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158747    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f16f01c-772c-4277-8519-8b017eceac4b" containerName="maas-api"
Oct 31 04:30:00.158980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158828    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0962c273-9b23-4c27-9e40-127a53cd190a" containerName="main"
Oct 31 04:30:00.158980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158840    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f16f01c-772c-4277-8519-8b017eceac4b" containerName="maas-api"
Oct 31 04:30:00.158980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158846    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0371044e-9bea-4b59-8329-2aa223d2c126" containerName="main"
Oct 31 04:30:00.158980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158853    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c308b1a-70f2-45b0-9a25-d2f8501a3b1c" containerName="main"
Oct 31 04:30:00.158980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158858    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="397fbdbf-8683-4d96-aa94-5e8ea77a785d" containerName="main"
Oct 31 04:30:00.158980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158865    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="66691fd9-2ce2-4311-a334-142046c0b848" containerName="istio-proxy"
Oct 31 04:30:00.158980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.158870    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e0f8b95-9b82-406c-8909-f8a11c81a753" containerName="main"
Oct 31 04:30:00.161661 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.161643    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"
Oct 31 04:30:00.252406 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.252366    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"]
Oct 31 04:30:00.282441 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.282400    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc015e4-70ca-473f-9e25-d01f0eb03c1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890350-wxsb4\" (UID: \"ebc015e4-70ca-473f-9e25-d01f0eb03c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"
Oct 31 04:30:00.282623 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.282453    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22zhs\" (UniqueName: \"kubernetes.io/projected/ebc015e4-70ca-473f-9e25-d01f0eb03c1d-kube-api-access-22zhs\") pod \"maas-api-key-cleanup-29890350-wxsb4\" (UID: \"ebc015e4-70ca-473f-9e25-d01f0eb03c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"
Oct 31 04:30:00.323533 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.323503    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s7gf9\""
Oct 31 04:30:00.383579 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.383537    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc015e4-70ca-473f-9e25-d01f0eb03c1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890350-wxsb4\" (UID: \"ebc015e4-70ca-473f-9e25-d01f0eb03c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"
Oct 31 04:30:00.383579 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.383584    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22zhs\" (UniqueName: \"kubernetes.io/projected/ebc015e4-70ca-473f-9e25-d01f0eb03c1d-kube-api-access-22zhs\") pod \"maas-api-key-cleanup-29890350-wxsb4\" (UID: \"ebc015e4-70ca-473f-9e25-d01f0eb03c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"
Oct 31 04:30:00.384293 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.384268    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc015e4-70ca-473f-9e25-d01f0eb03c1d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890350-wxsb4\" (UID: \"ebc015e4-70ca-473f-9e25-d01f0eb03c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"
Oct 31 04:30:00.440461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.440391    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22zhs\" (UniqueName: \"kubernetes.io/projected/ebc015e4-70ca-473f-9e25-d01f0eb03c1d-kube-api-access-22zhs\") pod \"maas-api-key-cleanup-29890350-wxsb4\" (UID: \"ebc015e4-70ca-473f-9e25-d01f0eb03c1d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"
Oct 31 04:30:00.475975 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.475940    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"
Oct 31 04:30:00.839397 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:30:00.839364    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc015e4_70ca_473f_9e25_d01f0eb03c1d.slice/crio-47a338e0f374df717a9e32e1183e3585c2198e4cd9f0123e5da1a469f0a7a2ad WatchSource:0}: Error finding container 47a338e0f374df717a9e32e1183e3585c2198e4cd9f0123e5da1a469f0a7a2ad: Status 404 returned error can't find the container with id 47a338e0f374df717a9e32e1183e3585c2198e4cd9f0123e5da1a469f0a7a2ad
Oct 31 04:30:00.841378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:00.841281    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"]
Oct 31 04:30:01.706306 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:01.706259    4900 generic.go:356] "Generic (PLEG): container finished" podID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d" containerID="3b42f55c5b8191e0098efdafa7e30fd22dd20a238284e3b9c8f03e64c33f04ee" exitCode=60
Oct 31 04:30:01.706717 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:01.706361    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4" event={"ID":"ebc015e4-70ca-473f-9e25-d01f0eb03c1d","Type":"ContainerDied","Data":"3b42f55c5b8191e0098efdafa7e30fd22dd20a238284e3b9c8f03e64c33f04ee"}
Oct 31 04:30:01.706717 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:01.706415    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4" event={"ID":"ebc015e4-70ca-473f-9e25-d01f0eb03c1d","Type":"ContainerStarted","Data":"47a338e0f374df717a9e32e1183e3585c2198e4cd9f0123e5da1a469f0a7a2ad"}
Oct 31 04:30:01.706717 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:01.706560    4900 scope.go:122] "RemoveContainer" containerID="3b42f55c5b8191e0098efdafa7e30fd22dd20a238284e3b9c8f03e64c33f04ee"
Oct 31 04:30:02.712909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:02.712875    4900 generic.go:356] "Generic (PLEG): container finished" podID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d" containerID="d7826dfceb1dbbb8bdee09008d5b2c0e1e398f9fbef713fb80dda5d5a1992916" exitCode=60
Oct 31 04:30:02.713411 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:02.712965    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4" event={"ID":"ebc015e4-70ca-473f-9e25-d01f0eb03c1d","Type":"ContainerDied","Data":"d7826dfceb1dbbb8bdee09008d5b2c0e1e398f9fbef713fb80dda5d5a1992916"}
Oct 31 04:30:02.713411 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:02.713039    4900 scope.go:122] "RemoveContainer" containerID="3b42f55c5b8191e0098efdafa7e30fd22dd20a238284e3b9c8f03e64c33f04ee"
Oct 31 04:30:02.713411 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:02.713254    4900 scope.go:122] "RemoveContainer" containerID="d7826dfceb1dbbb8bdee09008d5b2c0e1e398f9fbef713fb80dda5d5a1992916"
Oct 31 04:30:02.713591 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:30:02.713483    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890350-wxsb4_odh-ai-gateway-infra(ebc015e4-70ca-473f-9e25-d01f0eb03c1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4" podUID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d"
Oct 31 04:30:03.063251 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:03.063216    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:03.719466 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:03.719437    4900 scope.go:122] "RemoveContainer" containerID="d7826dfceb1dbbb8bdee09008d5b2c0e1e398f9fbef713fb80dda5d5a1992916"
Oct 31 04:30:03.719877 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:30:03.719652    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890350-wxsb4_odh-ai-gateway-infra(ebc015e4-70ca-473f-9e25-d01f0eb03c1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4" podUID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d"
Oct 31 04:30:06.983662 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:30:06.983627    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:30:07.883518 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:07.883477    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:10.400941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:10.400869    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:12.674718 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:12.674681    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:13.145410 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:13.145376    4900 scope.go:122] "RemoveContainer" containerID="d7826dfceb1dbbb8bdee09008d5b2c0e1e398f9fbef713fb80dda5d5a1992916"
Oct 31 04:30:13.764428 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:13.764394    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4" event={"ID":"ebc015e4-70ca-473f-9e25-d01f0eb03c1d","Type":"ContainerStarted","Data":"b8f75cf3a3ad59fc6c49d02d5b83ff47b047f6e8863d69b2d85460e97c088114"}
Oct 31 04:30:13.882557 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:13.882505    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4" podStartSLOduration=13.882492382 podStartE2EDuration="13.882492382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:30:13.880523352 +0000 UTC m=+4966.846856695" watchObservedRunningTime="2026-10-31 04:30:13.882492382 +0000 UTC m=+4966.848825737"
Oct 31 04:30:14.639181 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:14.639147    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:14.771252 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:14.771221    4900 generic.go:356] "Generic (PLEG): container finished" podID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d" containerID="b8f75cf3a3ad59fc6c49d02d5b83ff47b047f6e8863d69b2d85460e97c088114" exitCode=60
Oct 31 04:30:14.771701 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:14.771296    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4" event={"ID":"ebc015e4-70ca-473f-9e25-d01f0eb03c1d","Type":"ContainerDied","Data":"b8f75cf3a3ad59fc6c49d02d5b83ff47b047f6e8863d69b2d85460e97c088114"}
Oct 31 04:30:14.771701 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:14.771360    4900 scope.go:122] "RemoveContainer" containerID="d7826dfceb1dbbb8bdee09008d5b2c0e1e398f9fbef713fb80dda5d5a1992916"
Oct 31 04:30:14.771701 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:14.771687    4900 scope.go:122] "RemoveContainer" containerID="b8f75cf3a3ad59fc6c49d02d5b83ff47b047f6e8863d69b2d85460e97c088114"
Oct 31 04:30:14.772007 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:30:14.771978    4900 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890350-wxsb4_odh-ai-gateway-infra(ebc015e4-70ca-473f-9e25-d01f0eb03c1d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4" podUID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d"
Oct 31 04:30:14.846738 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:14.846709    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:14.891033 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:14.890972    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"]
Oct 31 04:30:15.911306 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:15.911281    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"
Oct 31 04:30:16.035457 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:16.035381    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebc015e4-70ca-473f-9e25-d01f0eb03c1d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc015e4-70ca-473f-9e25-d01f0eb03c1d-maas-api-service-ca\") pod \"ebc015e4-70ca-473f-9e25-d01f0eb03c1d\" (UID: \"ebc015e4-70ca-473f-9e25-d01f0eb03c1d\") "
Oct 31 04:30:16.035611 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:16.035490    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebc015e4-70ca-473f-9e25-d01f0eb03c1d-kube-api-access-22zhs\" (UniqueName: \"kubernetes.io/projected/ebc015e4-70ca-473f-9e25-d01f0eb03c1d-kube-api-access-22zhs\") pod \"ebc015e4-70ca-473f-9e25-d01f0eb03c1d\" (UID: \"ebc015e4-70ca-473f-9e25-d01f0eb03c1d\") "
Oct 31 04:30:16.035755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:16.035731    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc015e4-70ca-473f-9e25-d01f0eb03c1d-maas-api-service-ca" pod "ebc015e4-70ca-473f-9e25-d01f0eb03c1d" (UID: "ebc015e4-70ca-473f-9e25-d01f0eb03c1d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:30:16.037591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:16.037560    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc015e4-70ca-473f-9e25-d01f0eb03c1d-kube-api-access-22zhs" pod "ebc015e4-70ca-473f-9e25-d01f0eb03c1d" (UID: "ebc015e4-70ca-473f-9e25-d01f0eb03c1d"). InnerVolumeSpecName "kube-api-access-22zhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:30:16.136957 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:16.136921    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22zhs\" (UniqueName: \"kubernetes.io/projected/ebc015e4-70ca-473f-9e25-d01f0eb03c1d-kube-api-access-22zhs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:16.136957 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:16.136952    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebc015e4-70ca-473f-9e25-d01f0eb03c1d-maas-api-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:30:16.784688 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:16.784647    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4" event={"ID":"ebc015e4-70ca-473f-9e25-d01f0eb03c1d","Type":"ContainerDied","Data":"47a338e0f374df717a9e32e1183e3585c2198e4cd9f0123e5da1a469f0a7a2ad"}
Oct 31 04:30:16.784911 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:16.784691    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"
Oct 31 04:30:16.784911 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:16.784739    4900 scope.go:122] "RemoveContainer" containerID="b8f75cf3a3ad59fc6c49d02d5b83ff47b047f6e8863d69b2d85460e97c088114"
Oct 31 04:30:16.947424 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:16.947384    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"]
Oct 31 04:30:16.997324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:16.997287    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890350-wxsb4"]
Oct 31 04:30:17.153924 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:17.153839    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d" path="/var/lib/kubelet/pods/ebc015e4-70ca-473f-9e25-d01f0eb03c1d/volumes"
Oct 31 04:30:17.399097 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:17.399063    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:21.626052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:21.626013    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:23.141622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:23.141583    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:23.427957 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:23.427868    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:28.883490 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:28.883456    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:29.769353 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:29.769317    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:30:36.983649 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:30:36.983609    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:30:44.739126 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:30:44.739086    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:04.044113 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:04.044083    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"]
Oct 31 04:31:04.094084 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:04.094053    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-646f8765-qwqxb"]
Oct 31 04:31:05.151126 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:05.151095    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26655c2d-eaf1-482b-901f-8c52ec472f1d" path="/var/lib/kubelet/pods/26655c2d-eaf1-482b-901f-8c52ec472f1d/volumes"
Oct 31 04:31:06.982812 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:31:06.982762    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:31:10.507093 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:10.507057    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:10.732945 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:10.732912    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:12.548814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:12.548767    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:15.045622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:15.045585    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"]
Oct 31 04:31:15.095390 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:15.095355    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d57466af-srsjc"]
Oct 31 04:31:15.150689 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:15.150660    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b" path="/var/lib/kubelet/pods/33e1d05c-d6da-4ca6-bc56-ea8e9e893f9b/volumes"
Oct 31 04:31:22.270665 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.270625    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:22.678833 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.678742    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"]
Oct 31 04:31:22.679153 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.679139    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d" containerName="cleanup"
Oct 31 04:31:22.679197 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.679154    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d" containerName="cleanup"
Oct 31 04:31:22.679197 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.679179    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d" containerName="cleanup"
Oct 31 04:31:22.679197 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.679185    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d" containerName="cleanup"
Oct 31 04:31:22.679302 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.679245    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d" containerName="cleanup"
Oct 31 04:31:22.679302 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.679257    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d" containerName="cleanup"
Oct 31 04:31:22.682437 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.682421    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.776710 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.776678    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.776899 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.776724    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.776899 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.776844    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.777012 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.776905    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.777058 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.777038    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.777098 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.777074    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.777142 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.777100    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.777142 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.777123    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.777227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.777203    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-742tc\" (UniqueName: \"kubernetes.io/projected/ae6469aa-5349-41a7-9cb8-3ec92f92043c-kube-api-access-742tc\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.786370 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.786342    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"]
Oct 31 04:31:22.878150 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.878124    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.878288 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.878156    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.878288 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.878252    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.878404 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.878385    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.878467 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.878419    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.878467 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.878442    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.878578 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.878473    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-742tc\" (UniqueName: \"kubernetes.io/projected/ae6469aa-5349-41a7-9cb8-3ec92f92043c-kube-api-access-742tc\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.878578 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.878510    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.878578 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.878563    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.878735 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.878599    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-credential-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.878809 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.878772    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-workload-certs\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.878896 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.878877    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-workload-socket\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.878961 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.878904    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istio-data\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.879269 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.879242    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.880700 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.880668    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istio-envoy\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.881096 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.881075    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.935987 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.935932    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-742tc\" (UniqueName: \"kubernetes.io/projected/ae6469aa-5349-41a7-9cb8-3ec92f92043c-kube-api-access-742tc\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.935987 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.935954    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae6469aa-5349-41a7-9cb8-3ec92f92043c-istio-token\") pod \"maas-default-gateway-openshift-default-6886b96f9-9sp48\" (UID: \"ae6469aa-5349-41a7-9cb8-3ec92f92043c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:22.994438 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:22.994420    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:23.406022 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:31:23.405992    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6469aa_5349_41a7_9cb8_3ec92f92043c.slice/crio-8395da11730af5c4c4f9a8b866718a35ac4793e2684444d6861354848e9187d2 WatchSource:0}: Error finding container 8395da11730af5c4c4f9a8b866718a35ac4793e2684444d6861354848e9187d2: Status 404 returned error can't find the container with id 8395da11730af5c4c4f9a8b866718a35ac4793e2684444d6861354848e9187d2
Oct 31 04:31:23.407740 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:23.407719    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"]
Oct 31 04:31:23.408335 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:23.408305    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:31:23.408395 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:23.408373    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:31:23.408432 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:23.408416    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:31:24.093908 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:24.093868    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48" event={"ID":"ae6469aa-5349-41a7-9cb8-3ec92f92043c","Type":"ContainerStarted","Data":"53f08be579a1a851ed74bb6995e4167a67a652b2c741e3b0cc430f402479e655"}
Oct 31 04:31:24.094091 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:24.093950    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48" event={"ID":"ae6469aa-5349-41a7-9cb8-3ec92f92043c","Type":"ContainerStarted","Data":"8395da11730af5c4c4f9a8b866718a35ac4793e2684444d6861354848e9187d2"}
Oct 31 04:31:24.220564 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:24.220513    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48" podStartSLOduration=2.220500725 podStartE2EDuration="2.220500725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:31:24.219217211 +0000 UTC m=+5037.185550577" watchObservedRunningTime="2026-10-31 04:31:24.220500725 +0000 UTC m=+5037.186834068"
Oct 31 04:31:24.470883 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:24.470765    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:24.677727 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:24.677695    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:24.995114 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:24.995072    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:24.996769 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:24.996746    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9sp48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 04:31:24.996870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:24.996828    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48" podUID="ae6469aa-5349-41a7-9cb8-3ec92f92043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 04:31:25.995054 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:25.995020    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9sp48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 04:31:25.995447 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:25.995079    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48" podUID="ae6469aa-5349-41a7-9cb8-3ec92f92043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 04:31:26.994876 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:26.994840    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9sp48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 04:31:26.995060 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:26.994892    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48" podUID="ae6469aa-5349-41a7-9cb8-3ec92f92043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 04:31:27.995461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:27.995429    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9sp48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 04:31:27.995905 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:27.995491    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48" podUID="ae6469aa-5349-41a7-9cb8-3ec92f92043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 04:31:28.995477 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:28.995440    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9sp48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 04:31:28.995910 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:28.995494    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48" podUID="ae6469aa-5349-41a7-9cb8-3ec92f92043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 04:31:29.995351 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:29.995318    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9sp48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 04:31:29.995504 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:29.995369    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48" podUID="ae6469aa-5349-41a7-9cb8-3ec92f92043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 04:31:31.554157 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:31.554122    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:31.554631 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:31.554408    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:31.555453 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:31.555429    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48"
Oct 31 04:31:33.504197 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:33.504161    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:33.714758 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:31:33.714726    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:31:36.983644 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:31:36.983602    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:32:05.047073 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:32:05.047029    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"]
Oct 31 04:32:05.097767 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:32:05.097733    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbced6ba-bm8tv"]
Oct 31 04:32:05.150913 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:32:05.150875    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f38ad177-3298-48f2-9ec0-18a4a438a664" path="/var/lib/kubelet/pods/f38ad177-3298-48f2-9ec0-18a4a438a664/volumes"
Oct 31 04:32:06.983048 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:32:06.983015    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:32:11.120971 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:32:11.120939    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:32:29.015355 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:32:29.015323    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:32:29.018934 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:32:29.018909    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:32:29.288033 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:32:29.287941    4900 scope.go:122] "RemoveContainer" containerID="f6dd833bc62ee834dab576190885c7a2b9e4ce965fe3e71a13857b21b21e3059"
Oct 31 04:32:29.296410 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:32:29.296390    4900 scope.go:122] "RemoveContainer" containerID="dd9758c1b28209903758ae4f564eb9959fbacdb3631576199629e2d5b145cf32"
Oct 31 04:32:37.025804 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:32:37.025759    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:33:06.883186 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:06.883151    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"]
Oct 31 04:33:06.883657 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:06.883570    4900 cpu_manager.go:414] "containerMap: removing container" podUID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d" containerName="cleanup"
Oct 31 04:33:06.883657 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:06.883584    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d" containerName="cleanup"
Oct 31 04:33:06.883657 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:06.883655    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebc015e4-70ca-473f-9e25-d01f0eb03c1d" containerName="cleanup"
Oct 31 04:33:06.886666 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:06.886650    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:06.939598 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:06.939552    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"]
Oct 31 04:33:06.940509 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:06.940480    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-dshm\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:06.940738 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:06.940716    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-model-cache\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:06.940910 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:06.940895    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-tmp-dir\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:06.941002 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:06.940989    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-home\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:06.941066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:06.941035    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vrk\" (UniqueName: \"kubernetes.io/projected/8daa85e0-82a5-487b-9271-1dce5f5e9191-kube-api-access-k9vrk\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:06.941123 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:06.941097    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8daa85e0-82a5-487b-9271-1dce5f5e9191-tls-certs\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:06.982854 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:33:06.982814    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:33:06.987753 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:06.987735    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-7010fecc-kserve-self-signed-certs\""
Oct 31 04:33:07.042299 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.042191    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8daa85e0-82a5-487b-9271-1dce5f5e9191-tls-certs\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:07.042299 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.042275    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-dshm\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:07.042643 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.042317    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-model-cache\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:07.042643 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.042422    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-tmp-dir\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:07.042643 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.042622    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-home\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:07.043339 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.043291    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vrk\" (UniqueName: \"kubernetes.io/projected/8daa85e0-82a5-487b-9271-1dce5f5e9191-kube-api-access-k9vrk\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:07.043808 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.043767    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-model-cache\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:07.043934 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.043860    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-tmp-dir\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:07.044095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.044068    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-home\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:07.044999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.044976    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-dshm\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:07.045450 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.045430    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8daa85e0-82a5-487b-9271-1dce5f5e9191-tls-certs\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:07.101184 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.101152    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vrk\" (UniqueName: \"kubernetes.io/projected/8daa85e0-82a5-487b-9271-1dce5f5e9191-kube-api-access-k9vrk\") pod \"e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") " pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:07.106951 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.106926    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:07.197673 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.197590    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:07.564508 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:33:07.564475    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8daa85e0_82a5_487b_9271_1dce5f5e9191.slice/crio-a8444ac7ceb8e31453db4467e7f1450d4f3fe05a9467f5803a0228339fa26cdd WatchSource:0}: Error finding container a8444ac7ceb8e31453db4467e7f1450d4f3fe05a9467f5803a0228339fa26cdd: Status 404 returned error can't find the container with id a8444ac7ceb8e31453db4467e7f1450d4f3fe05a9467f5803a0228339fa26cdd
Oct 31 04:33:07.566296 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.566272    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"]
Oct 31 04:33:07.641096 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.641063    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl" event={"ID":"8daa85e0-82a5-487b-9271-1dce5f5e9191","Type":"ContainerStarted","Data":"1777efd2ea3bd69e3571ecfd0ca275994ff00b2e8b2100e6f6c14f2abf67d8ec"}
Oct 31 04:33:07.641293 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.641103    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl" event={"ID":"8daa85e0-82a5-487b-9271-1dce5f5e9191","Type":"ContainerStarted","Data":"a8444ac7ceb8e31453db4467e7f1450d4f3fe05a9467f5803a0228339fa26cdd"}
Oct 31 04:33:07.755390 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:07.755337    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl" podStartSLOduration=1.75532427 podStartE2EDuration="1.75532427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:33:07.755112389 +0000 UTC m=+5140.721445733" watchObservedRunningTime="2026-10-31 04:33:07.75532427 +0000 UTC m=+5140.721657613"
Oct 31 04:33:08.198250 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:08.198218    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:08.945138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:08.945092    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:10.530337 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:10.530279    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:10.785046 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:10.784952    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:11.436901 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:11.436869    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:33:17.204812 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:17.203854    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:17.226626 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:17.226594    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:17.598812 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:17.595905    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9sp48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:33:17.598812 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:17.595994    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48" podUID="ae6469aa-5349-41a7-9cb8-3ec92f92043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:33:17.719185 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:17.719154    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:33:29.318514 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:33:29.318479    4900 scope.go:122] "RemoveContainer" containerID="f865a90fb21b88197b8bf82610ad79372167aa38d85ed64a33177b7505c41435"
Oct 31 04:33:36.984136 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:33:36.984094    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:34:04.446722 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:04.446686    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:04.649719 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:04.649687    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:06.984073 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:34:06.984033    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:34:11.771052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:11.771019    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:18.060682 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:18.060637    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"]
Oct 31 04:34:18.141623 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:18.141591    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:18.356219 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:18.356134    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:25.610016 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:25.609971    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lxq9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:34:25.612556 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:25.610561    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" podUID="d7f8ce75-4372-4c11-aea6-6922ed91ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:34:29.338180 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:29.338146    4900 scope.go:122] "RemoveContainer" containerID="f3d3e94912cc9a03f12fcba39f2240bfb85d19a1978e4f59c50a6f39fd53a0de"
Oct 31 04:34:33.081483 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.081413    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl" podUID="8daa85e0-82a5-487b-9271-1dce5f5e9191" containerName="main" containerID="cri-o://1777efd2ea3bd69e3571ecfd0ca275994ff00b2e8b2100e6f6c14f2abf67d8ec" gracePeriod=45
Oct 31 04:34:33.337323 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.337256    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:34:33.516037 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.516003    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-tmp-dir\") pod \"8daa85e0-82a5-487b-9271-1dce5f5e9191\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") "
Oct 31 04:34:33.516244 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.516107    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8daa85e0-82a5-487b-9271-1dce5f5e9191-tls-certs\" (UniqueName: \"kubernetes.io/secret/8daa85e0-82a5-487b-9271-1dce5f5e9191-tls-certs\") pod \"8daa85e0-82a5-487b-9271-1dce5f5e9191\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") "
Oct 31 04:34:33.516244 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.516131    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8daa85e0-82a5-487b-9271-1dce5f5e9191-kube-api-access-k9vrk\" (UniqueName: \"kubernetes.io/projected/8daa85e0-82a5-487b-9271-1dce5f5e9191-kube-api-access-k9vrk\") pod \"8daa85e0-82a5-487b-9271-1dce5f5e9191\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") "
Oct 31 04:34:33.516244 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.516161    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-model-cache\") pod \"8daa85e0-82a5-487b-9271-1dce5f5e9191\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") "
Oct 31 04:34:33.516244 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.516203    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-dshm\") pod \"8daa85e0-82a5-487b-9271-1dce5f5e9191\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") "
Oct 31 04:34:33.516244 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.516236    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-home\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-home\") pod \"8daa85e0-82a5-487b-9271-1dce5f5e9191\" (UID: \"8daa85e0-82a5-487b-9271-1dce5f5e9191\") "
Oct 31 04:34:33.516543 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.516310    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-tmp-dir" pod "8daa85e0-82a5-487b-9271-1dce5f5e9191" (UID: "8daa85e0-82a5-487b-9271-1dce5f5e9191"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:34:33.516543 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.516473    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-model-cache" pod "8daa85e0-82a5-487b-9271-1dce5f5e9191" (UID: "8daa85e0-82a5-487b-9271-1dce5f5e9191"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:34:33.516543 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.516489    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-home" pod "8daa85e0-82a5-487b-9271-1dce5f5e9191" (UID: "8daa85e0-82a5-487b-9271-1dce5f5e9191"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:34:33.516673 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.516648    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:34:33.516719 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.516679    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:34:33.516719 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.516694    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:34:33.518496 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.518461    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-dshm" pod "8daa85e0-82a5-487b-9271-1dce5f5e9191" (UID: "8daa85e0-82a5-487b-9271-1dce5f5e9191"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:34:33.518496 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.518468    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8daa85e0-82a5-487b-9271-1dce5f5e9191-tls-certs" pod "8daa85e0-82a5-487b-9271-1dce5f5e9191" (UID: "8daa85e0-82a5-487b-9271-1dce5f5e9191"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:34:33.518690 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.518523    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8daa85e0-82a5-487b-9271-1dce5f5e9191-kube-api-access-k9vrk" pod "8daa85e0-82a5-487b-9271-1dce5f5e9191" (UID: "8daa85e0-82a5-487b-9271-1dce5f5e9191"). InnerVolumeSpecName "kube-api-access-k9vrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:34:33.617690 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.617604    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8daa85e0-82a5-487b-9271-1dce5f5e9191-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:34:33.617690 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.617638    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9vrk\" (UniqueName: \"kubernetes.io/projected/8daa85e0-82a5-487b-9271-1dce5f5e9191-kube-api-access-k9vrk\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:34:33.617690 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:33.617649    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8daa85e0-82a5-487b-9271-1dce5f5e9191-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:34:34.066401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:34.066364    4900 generic.go:356] "Generic (PLEG): container finished" podID="8daa85e0-82a5-487b-9271-1dce5f5e9191" containerID="1777efd2ea3bd69e3571ecfd0ca275994ff00b2e8b2100e6f6c14f2abf67d8ec" exitCode=0
Oct 31 04:34:34.066593 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:34.066407    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"
Oct 31 04:34:34.066593 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:34.066416    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl" event={"ID":"8daa85e0-82a5-487b-9271-1dce5f5e9191","Type":"ContainerDied","Data":"1777efd2ea3bd69e3571ecfd0ca275994ff00b2e8b2100e6f6c14f2abf67d8ec"}
Oct 31 04:34:34.066593 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:34.066446    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl" event={"ID":"8daa85e0-82a5-487b-9271-1dce5f5e9191","Type":"ContainerDied","Data":"a8444ac7ceb8e31453db4467e7f1450d4f3fe05a9467f5803a0228339fa26cdd"}
Oct 31 04:34:34.066593 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:34.066463    4900 scope.go:122] "RemoveContainer" containerID="1777efd2ea3bd69e3571ecfd0ca275994ff00b2e8b2100e6f6c14f2abf67d8ec"
Oct 31 04:34:34.075660 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:34.075639    4900 scope.go:122] "RemoveContainer" containerID="1777efd2ea3bd69e3571ecfd0ca275994ff00b2e8b2100e6f6c14f2abf67d8ec"
Oct 31 04:34:34.076022 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:34:34.076000    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1777efd2ea3bd69e3571ecfd0ca275994ff00b2e8b2100e6f6c14f2abf67d8ec\": container with ID starting with 1777efd2ea3bd69e3571ecfd0ca275994ff00b2e8b2100e6f6c14f2abf67d8ec not found: ID does not exist" containerID="1777efd2ea3bd69e3571ecfd0ca275994ff00b2e8b2100e6f6c14f2abf67d8ec"
Oct 31 04:34:34.076105 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:34.076030    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1777efd2ea3bd69e3571ecfd0ca275994ff00b2e8b2100e6f6c14f2abf67d8ec"} err="failed to get container status \"1777efd2ea3bd69e3571ecfd0ca275994ff00b2e8b2100e6f6c14f2abf67d8ec\": rpc error: code = NotFound desc = could not find container \"1777efd2ea3bd69e3571ecfd0ca275994ff00b2e8b2100e6f6c14f2abf67d8ec\": container with ID starting with 1777efd2ea3bd69e3571ecfd0ca275994ff00b2e8b2100e6f6c14f2abf67d8ec not found: ID does not exist"
Oct 31 04:34:34.247139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:34.247102    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"]
Oct 31 04:34:34.265414 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:34.265383    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-7010fecc-kserve-587655c754-zr7gl"]
Oct 31 04:34:35.150877 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:35.150844    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8daa85e0-82a5-487b-9271-1dce5f5e9191" path="/var/lib/kubelet/pods/8daa85e0-82a5-487b-9271-1dce5f5e9191/volumes"
Oct 31 04:34:36.078434 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.078387    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:36.539745 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.539713    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"]
Oct 31 04:34:36.540246 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.540227    4900 cpu_manager.go:414] "containerMap: removing container" podUID="8daa85e0-82a5-487b-9271-1dce5f5e9191" containerName="main"
Oct 31 04:34:36.540352 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.540249    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8daa85e0-82a5-487b-9271-1dce5f5e9191" containerName="main"
Oct 31 04:34:36.540413 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.540363    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="8daa85e0-82a5-487b-9271-1dce5f5e9191" containerName="main"
Oct 31 04:34:36.545039 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.545018    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.545554 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.545532    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwn5l\" (UniqueName: \"kubernetes.io/projected/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-kube-api-access-xwn5l\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.545657 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.545640    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-data\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.545702 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.545678    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-envoy\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.545736 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.545703    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-credential-socket\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.545815 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.545803    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-podinfo\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.545860 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.545835    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-workload-certs\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.545916 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.545894    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-token\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.545980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.545969    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istiod-ca-cert\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.546018 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.545998    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-workload-socket\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.598797 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.598762    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-bbc981-openshift-default-dockercfg-77jnc\""
Oct 31 04:34:36.646652 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.646614    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istiod-ca-cert\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.646652 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.646655    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-workload-socket\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.646938 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.646803    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwn5l\" (UniqueName: \"kubernetes.io/projected/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-kube-api-access-xwn5l\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.646938 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.646907    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-data\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.647068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.646953    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-envoy\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.647068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.646989    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-credential-socket\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.647068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.647022    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-podinfo\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.647068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.647036    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-workload-socket\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.647068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.647048    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-workload-certs\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.647341 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.647222    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-token\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.647341 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.647255    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-data\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.647499 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.647480    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istiod-ca-cert\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.647672 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.647650    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-credential-socket\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.647756 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.647668    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-workload-certs\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.649170 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.649142    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-envoy\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.649606 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.649588    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-podinfo\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.651552 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.651530    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"]
Oct 31 04:34:36.706177 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.706147    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-token\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.706364 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.706346    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwn5l\" (UniqueName: \"kubernetes.io/projected/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-kube-api-access-xwn5l\") pod \"e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") " pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.729556 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.729531    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:36.858212 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.858126    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:36.939725 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:36.939688    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:36.984043 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:34:36.984009    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:34:37.272827 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:34:37.272759    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb90bf4f_d30f_4af7_8bdf_355a2a3b0993.slice/crio-0f6bc945493efd16d3d33eb673c73545ba185d42b73b3af2734ec78c2fd37318 WatchSource:0}: Error finding container 0f6bc945493efd16d3d33eb673c73545ba185d42b73b3af2734ec78c2fd37318: Status 404 returned error can't find the container with id 0f6bc945493efd16d3d33eb673c73545ba185d42b73b3af2734ec78c2fd37318
Oct 31 04:34:37.274439 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:37.274413    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"]
Oct 31 04:34:37.275198 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:37.275167    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:34:37.275319 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:37.275240    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:34:37.275319 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:37.275281    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:34:38.086470 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:38.086434    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc" event={"ID":"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993","Type":"ContainerStarted","Data":"d0f018d4aae2604ad64c176f5348a173be1773621d5bceba42757daf1ebee6b9"}
Oct 31 04:34:38.086693 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:38.086488    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc" event={"ID":"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993","Type":"ContainerStarted","Data":"0f6bc945493efd16d3d33eb673c73545ba185d42b73b3af2734ec78c2fd37318"}
Oct 31 04:34:38.253753 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:38.253693    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc" podStartSLOduration=2.253674678 podStartE2EDuration="2.253674678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:38.252781861 +0000 UTC m=+5231.219115204" watchObservedRunningTime="2026-10-31 04:34:38.253674678 +0000 UTC m=+5231.220008023"
Oct 31 04:34:38.858614 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:38.858578    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:38.863445 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:38.863416    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:39.090838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:39.090751    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:39.091855 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:39.091823    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:34:41.306461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:41.306413    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:41.535270 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:41.535232    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:44.582915 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:44.582878    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:44.799333 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:44.799295    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:44.920396 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:44.920306    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"]
Oct 31 04:34:44.923768 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:44.923749    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"
Oct 31 04:34:44.953229 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:44.953191    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a8662f-7bc5-42c5-811b-c309968e090b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4\" (UID: \"56a8662f-7bc5-42c5-811b-c309968e090b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"
Oct 31 04:34:44.953427 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:44.953346    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7pdm\" (UniqueName: \"kubernetes.io/projected/56a8662f-7bc5-42c5-811b-c309968e090b-kube-api-access-z7pdm\") pod \"payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4\" (UID: \"56a8662f-7bc5-42c5-811b-c309968e090b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"
Oct 31 04:34:44.976628 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:44.976595    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"]
Oct 31 04:34:45.024739 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.024706    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-bbc981\""
Oct 31 04:34:45.054711 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.054672    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a8662f-7bc5-42c5-811b-c309968e090b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4\" (UID: \"56a8662f-7bc5-42c5-811b-c309968e090b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"
Oct 31 04:34:45.054946 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.054866    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7pdm\" (UniqueName: \"kubernetes.io/projected/56a8662f-7bc5-42c5-811b-c309968e090b-kube-api-access-z7pdm\") pod \"payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4\" (UID: \"56a8662f-7bc5-42c5-811b-c309968e090b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"
Oct 31 04:34:45.055589 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.055559    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a8662f-7bc5-42c5-811b-c309968e090b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4\" (UID: \"56a8662f-7bc5-42c5-811b-c309968e090b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"
Oct 31 04:34:45.065709 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.065676    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"]
Oct 31 04:34:45.070537 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.070504    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"
Oct 31 04:34:45.112168 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.112132    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7pdm\" (UniqueName: \"kubernetes.io/projected/56a8662f-7bc5-42c5-811b-c309968e090b-kube-api-access-z7pdm\") pod \"payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4\" (UID: \"56a8662f-7bc5-42c5-811b-c309968e090b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"
Oct 31 04:34:45.123240 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.123205    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"]
Oct 31 04:34:45.134649 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.134615    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-dfbdce6e-dockercfg-mw6qc\""
Oct 31 04:34:45.156376 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.156334    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9042e89-be27-4be6-9820-b5bea3a61a9d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr\" (UID: \"f9042e89-be27-4be6-9820-b5bea3a61a9d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"
Oct 31 04:34:45.156561 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.156448    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk5df\" (UniqueName: \"kubernetes.io/projected/f9042e89-be27-4be6-9820-b5bea3a61a9d-kube-api-access-wk5df\") pod \"payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr\" (UID: \"f9042e89-be27-4be6-9820-b5bea3a61a9d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"
Oct 31 04:34:45.237178 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.237147    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"
Oct 31 04:34:45.257755 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.257707    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9042e89-be27-4be6-9820-b5bea3a61a9d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr\" (UID: \"f9042e89-be27-4be6-9820-b5bea3a61a9d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"
Oct 31 04:34:45.257921 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.257810    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk5df\" (UniqueName: \"kubernetes.io/projected/f9042e89-be27-4be6-9820-b5bea3a61a9d-kube-api-access-wk5df\") pod \"payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr\" (UID: \"f9042e89-be27-4be6-9820-b5bea3a61a9d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"
Oct 31 04:34:45.258543 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.258523    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9042e89-be27-4be6-9820-b5bea3a61a9d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr\" (UID: \"f9042e89-be27-4be6-9820-b5bea3a61a9d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"
Oct 31 04:34:45.314810 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.314737    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk5df\" (UniqueName: \"kubernetes.io/projected/f9042e89-be27-4be6-9820-b5bea3a61a9d-kube-api-access-wk5df\") pod \"payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr\" (UID: \"f9042e89-be27-4be6-9820-b5bea3a61a9d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"
Oct 31 04:34:45.387666 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.387627    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"
Oct 31 04:34:45.602773 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:34:45.602743    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a8662f_7bc5_42c5_811b_c309968e090b.slice/crio-579e1051d65d9fbc86b99f7153b75a1b5fe6e73138c725077f126fdee7965391 WatchSource:0}: Error finding container 579e1051d65d9fbc86b99f7153b75a1b5fe6e73138c725077f126fdee7965391: Status 404 returned error can't find the container with id 579e1051d65d9fbc86b99f7153b75a1b5fe6e73138c725077f126fdee7965391
Oct 31 04:34:45.604567 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.604539    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"]
Oct 31 04:34:45.752663 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:34:45.752626    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9042e89_be27_4be6_9820_b5bea3a61a9d.slice/crio-43d00c71813fa563f6652c36263d2ae313b1a63f233e418166972d0ecda49438 WatchSource:0}: Error finding container 43d00c71813fa563f6652c36263d2ae313b1a63f233e418166972d0ecda49438: Status 404 returned error can't find the container with id 43d00c71813fa563f6652c36263d2ae313b1a63f233e418166972d0ecda49438
Oct 31 04:34:45.754078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:45.754052    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"]
Oct 31 04:34:46.145352 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:46.145312    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4" event={"ID":"56a8662f-7bc5-42c5-811b-c309968e090b","Type":"ContainerStarted","Data":"356105836e4ed892e38714f211b43e14b5c1332f9a351895aaf2a9b32eac0bcb"}
Oct 31 04:34:46.145574 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:46.145370    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4" event={"ID":"56a8662f-7bc5-42c5-811b-c309968e090b","Type":"ContainerStarted","Data":"579e1051d65d9fbc86b99f7153b75a1b5fe6e73138c725077f126fdee7965391"}
Oct 31 04:34:46.145574 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:46.145494    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"
Oct 31 04:34:46.147007 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:46.146977    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr" event={"ID":"f9042e89-be27-4be6-9820-b5bea3a61a9d","Type":"ContainerStarted","Data":"f13e2c48ecd14799e05097f6b0e1ece8588b6366a992d0f7973a680804f0513a"}
Oct 31 04:34:46.147168 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:46.147027    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"
Oct 31 04:34:46.147168 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:46.147051    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr" event={"ID":"f9042e89-be27-4be6-9820-b5bea3a61a9d","Type":"ContainerStarted","Data":"43d00c71813fa563f6652c36263d2ae313b1a63f233e418166972d0ecda49438"}
Oct 31 04:34:46.259158 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:46.259106    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4" podStartSLOduration=2.259092399 podStartE2EDuration="2.259092399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:46.257621022 +0000 UTC m=+5239.223954366" watchObservedRunningTime="2026-10-31 04:34:46.259092399 +0000 UTC m=+5239.225425742"
Oct 31 04:34:46.373111 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:46.372692    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr" podStartSLOduration=1.372672745 podStartE2EDuration="1.372672745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:46.370226312 +0000 UTC m=+5239.336559650" watchObservedRunningTime="2026-10-31 04:34:46.372672745 +0000 UTC m=+5239.339006091"
Oct 31 04:34:47.247803 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.247753    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"]
Oct 31 04:34:47.251529 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.251508    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:47.353878 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.353845    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-bbc981-serving-cert\""
Oct 31 04:34:47.355633 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.355611    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"]
Oct 31 04:34:47.381414 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.381374    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:47.381615 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.381512    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6019750-4137-48bc-a9f3-7cf5c8819415-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:47.381615 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.381570    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:47.381802 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.381767    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfzjn\" (UniqueName: \"kubernetes.io/projected/c6019750-4137-48bc-a9f3-7cf5c8819415-kube-api-access-qfzjn\") pod \"maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:47.482837 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.482797    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfzjn\" (UniqueName: \"kubernetes.io/projected/c6019750-4137-48bc-a9f3-7cf5c8819415-kube-api-access-qfzjn\") pod \"maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:47.483053 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.482876    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:47.483053 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.482972    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6019750-4137-48bc-a9f3-7cf5c8819415-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:47.483053 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.483018    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:47.483053 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:34:47.483030    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-serving-cert: secret "maas-api-e2e-worker-single-main-bbc981-serving-cert" not found
Oct 31 04:34:47.483351 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:34:47.483103    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-maas-api-tls podName:c6019750-4137-48bc-a9f3-7cf5c8819415 nodeName:}" failed. No retries permitted until 2026-10-31 04:34:47.983081908 +0000 UTC m=+5240.949415250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-maas-api-tls") pod "maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r" (UID: "c6019750-4137-48bc-a9f3-7cf5c8819415") : secret "maas-api-e2e-worker-single-main-bbc981-serving-cert" not found
Oct 31 04:34:47.483870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.483842    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6019750-4137-48bc-a9f3-7cf5c8819415-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:47.486286 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.486259    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-metrics-certs\") pod \"maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:47.505984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.505890    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:47.540521 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.540486    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfzjn\" (UniqueName: \"kubernetes.io/projected/c6019750-4137-48bc-a9f3-7cf5c8819415-kube-api-access-qfzjn\") pod \"maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:47.703004 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.702956    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:47.989867 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.989835    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:47.992300 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:47.992273    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:48.164762 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:48.164730    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:48.533099 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:34:48.533063    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6019750_4137_48bc_a9f3_7cf5c8819415.slice/crio-26ee80aa0b02dbf3a0dc5e03a0fc98a97128ee50cec812b5a0f3143ba749aa07 WatchSource:0}: Error finding container 26ee80aa0b02dbf3a0dc5e03a0fc98a97128ee50cec812b5a0f3143ba749aa07: Status 404 returned error can't find the container with id 26ee80aa0b02dbf3a0dc5e03a0fc98a97128ee50cec812b5a0f3143ba749aa07
Oct 31 04:34:48.533882 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:48.533846    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"]
Oct 31 04:34:48.534540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:48.534523    4900 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:34:49.165883 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:49.165843    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r" event={"ID":"c6019750-4137-48bc-a9f3-7cf5c8819415","Type":"ContainerStarted","Data":"26ee80aa0b02dbf3a0dc5e03a0fc98a97128ee50cec812b5a0f3143ba749aa07"}
Oct 31 04:34:50.172440 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:50.172397    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r" event={"ID":"c6019750-4137-48bc-a9f3-7cf5c8819415","Type":"ContainerStarted","Data":"2731c1e27ff7a8c094ae543039809bd0ac02c705620c5b584b68ebaaf7cfcbe6"}
Oct 31 04:34:50.172962 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:50.172461    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:50.286742 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:50.286684    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r" podStartSLOduration=2.317127814 podStartE2EDuration="3.286669547s" totalImagesPullingTime="969.541733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:34:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:34:50.285552942 +0000 UTC m=+5243.251886286" watchObservedRunningTime="2026-10-31 04:34:50.286669547 +0000 UTC m=+5243.253002891"
Oct 31 04:34:52.113347 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:52.113309    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:56.188930 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:56.188902    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:34:56.659807 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:56.659756    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:34:57.166425 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:57.166393    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"
Oct 31 04:34:57.166623 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:34:57.166470    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"
Oct 31 04:35:01.164519 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:01.163361    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:04.347459 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:04.347421    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:04.607284 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:04.607199    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:05.242595 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:05.242564    4900 generic.go:356] "Generic (PLEG): container finished" podID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" containerID="d0f018d4aae2604ad64c176f5348a173be1773621d5bceba42757daf1ebee6b9" exitCode=0
Oct 31 04:35:05.242844 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:05.242634    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc" event={"ID":"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993","Type":"ContainerDied","Data":"d0f018d4aae2604ad64c176f5348a173be1773621d5bceba42757daf1ebee6b9"}
Oct 31 04:35:05.243053 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:05.243039    4900 scope.go:122] "RemoveContainer" containerID="d0f018d4aae2604ad64c176f5348a173be1773621d5bceba42757daf1ebee6b9"
Oct 31 04:35:05.244250 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:05.244223    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:35:05.244319 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:05.244306    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:35:05.244361 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:05.244340    4900 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:35:05.850384 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:05.849680    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:06.058813 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:06.056116    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:06.264961 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:06.264923    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc" event={"ID":"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993","Type":"ContainerStarted","Data":"51ffeba5427586589204dd442e94a88abe319a3f883b3cf3288c6304a3d18c4b"}
Oct 31 04:35:06.624284 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:06.624196    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:06.848268 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:06.848143    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:06.859012 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:06.858974    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:35:06.860222 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:06.860192    4900 patch_prober.go:29] interesting pod/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 04:35:06.860361 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:06.860245    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc" podUID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 04:35:06.986229 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:35:06.986195    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:35:07.859314 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:07.859262    4900 patch_prober.go:29] interesting pod/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 04:35:07.859750 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:07.859335    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc" podUID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 04:35:08.859247 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:08.859208    4900 patch_prober.go:29] interesting pod/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 04:35:08.859470 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:08.859281    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc" podUID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 04:35:09.091549 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.091514    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:35:09.621826 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.621773    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"]
Oct 31 04:35:09.624547 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.624526    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.726875 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.726841    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-bbc981\"/\"kube-root-ca.crt\""
Oct 31 04:35:09.726875 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.726867    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-bbc981\"/\"simulator-single-main-bbc981-kserve-self-signed-certs\""
Oct 31 04:35:09.727167 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.726952    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-bbc981\"/\"default-dockercfg-d54sd\""
Oct 31 04:35:09.727167 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.726988    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-bbc981\"/\"openshift-service-ca.crt\""
Oct 31 04:35:09.727289 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.727232    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e57a668a-c6a5-492d-b2f5-c9e8693fb054-tls-certs\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.727289 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.727275    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d687v\" (UniqueName: \"kubernetes.io/projected/e57a668a-c6a5-492d-b2f5-c9e8693fb054-kube-api-access-d687v\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.727438 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.727414    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-dshm\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.727503 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.727459    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-home\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.727579 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.727564    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-tmp-dir\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.727636 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.727593    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-model-cache\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.728930 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.728913    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"]
Oct 31 04:35:09.791014 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.790965    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:09.829322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.829289    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e57a668a-c6a5-492d-b2f5-c9e8693fb054-tls-certs\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.829322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.829325    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d687v\" (UniqueName: \"kubernetes.io/projected/e57a668a-c6a5-492d-b2f5-c9e8693fb054-kube-api-access-d687v\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.829566 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.829372    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-dshm\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.829566 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.829391    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-home\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.829566 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.829460    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-tmp-dir\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.829566 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.829489    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-model-cache\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.829902 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.829876    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-home\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.829992 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.829918    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-tmp-dir\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.829992 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.829940    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-model-cache\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.831730 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.831709    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-dshm\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.831932 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.831915    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e57a668a-c6a5-492d-b2f5-c9e8693fb054-tls-certs\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.859368 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.859333    4900 patch_prober.go:29] interesting pod/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 04:35:09.859651 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.859621    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc" podUID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 04:35:09.886189 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.886115    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d687v\" (UniqueName: \"kubernetes.io/projected/e57a668a-c6a5-492d-b2f5-c9e8693fb054-kube-api-access-d687v\") pod \"simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") " pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:09.954289 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:09.954000    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:10.321812 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:10.321755    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"]
Oct 31 04:35:10.322174 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:35:10.322148    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode57a668a_c6a5_492d_b2f5_c9e8693fb054.slice/crio-b217693323c44d20e2138900168ebe5b83753255d437a073249cadc454864768 WatchSource:0}: Error finding container b217693323c44d20e2138900168ebe5b83753255d437a073249cadc454864768: Status 404 returned error can't find the container with id b217693323c44d20e2138900168ebe5b83753255d437a073249cadc454864768
Oct 31 04:35:10.366053 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:10.365134    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:10.859686 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:10.859195    4900 patch_prober.go:29] interesting pod/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 04:35:10.859686 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:10.859260    4900 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc" podUID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 04:35:11.289927 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:11.289888    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s" event={"ID":"e57a668a-c6a5-492d-b2f5-c9e8693fb054","Type":"ContainerStarted","Data":"127094a38e65da2b992fd7e1e1aecb327c54dddded118f36327158859664661a"}
Oct 31 04:35:11.289927 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:11.289932    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s" event={"ID":"e57a668a-c6a5-492d-b2f5-c9e8693fb054","Type":"ContainerStarted","Data":"b217693323c44d20e2138900168ebe5b83753255d437a073249cadc454864768"}
Oct 31 04:35:11.404090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:11.404031    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s" podStartSLOduration=2.404016836 podStartE2EDuration="2.404016836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:35:11.403005903 +0000 UTC m=+5264.369339244" watchObservedRunningTime="2026-10-31 04:35:11.404016836 +0000 UTC m=+5264.370350180"
Oct 31 04:35:11.862715 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:11.862686    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:35:11.863606 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:11.863589    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:35:11.955694 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:11.955657    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:12.119702 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:12.119618    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:12.323777 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:12.323740    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:14.842471 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:14.842439    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:19.230147 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:19.230103    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:19.954966 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:19.954926    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:19.967935 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:19.967905    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:20.349830 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:20.349775    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:35:23.745075 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:23.745042    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:28.148536 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:28.148487    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:32.634577 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:32.634539    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:36.983692 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:35:36.983653    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:35:37.017250 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:37.017210    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:41.561929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:41.561895    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:45.931187 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:45.931155    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:50.430061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:50.430015    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:54.850806 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:54.850746    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:35:59.323594 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:35:59.323543    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:03.731494 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:03.731460    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:05.567061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:05.567026    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:06.001110 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:06.001078    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:06.984533 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:36:06.984497    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:36:08.144392 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:08.144358    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:10.454853 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:10.454643    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"]
Oct 31 04:36:10.598615 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:10.598582    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:11.761901 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:11.761862    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:12.458328 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:12.458291    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:12.465585 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:12.465557    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"]
Oct 31 04:36:12.574621 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:12.574588    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"]
Oct 31 04:36:12.663688 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:12.663656    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:12.911361 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:12.911325    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:13.115430 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:13.115395    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:16.595216 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:16.595180    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"]
Oct 31 04:36:16.595675 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:16.595401    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r" podUID="c6019750-4137-48bc-a9f3-7cf5c8819415" containerName="maas-api" containerID="cri-o://2731c1e27ff7a8c094ae543039809bd0ac02c705620c5b584b68ebaaf7cfcbe6" gracePeriod=30
Oct 31 04:36:16.771062 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:16.771029    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:17.466568 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.466484    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr" podUID="f9042e89-be27-4be6-9820-b5bea3a61a9d" containerName="payload-processing" containerID="cri-o://f13e2c48ecd14799e05097f6b0e1ece8588b6366a992d0f7973a680804f0513a" gracePeriod=25
Oct 31 04:36:17.513769 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.513733    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:17.575813 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.575728    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4" podUID="56a8662f-7bc5-42c5-811b-c309968e090b" containerName="payload-pre-processing" containerID="cri-o://356105836e4ed892e38714f211b43e14b5c1332f9a351895aaf2a9b32eac0bcb" gracePeriod=25
Oct 31 04:36:17.844256 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.844232    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:36:17.879232 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.879197    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6019750-4137-48bc-a9f3-7cf5c8819415-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6019750-4137-48bc-a9f3-7cf5c8819415-openshift-service-ca\") pod \"c6019750-4137-48bc-a9f3-7cf5c8819415\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") "
Oct 31 04:36:17.879394 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.879313    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-maas-api-tls\") pod \"c6019750-4137-48bc-a9f3-7cf5c8819415\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") "
Oct 31 04:36:17.879394 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.879345    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6019750-4137-48bc-a9f3-7cf5c8819415-kube-api-access-qfzjn\" (UniqueName: \"kubernetes.io/projected/c6019750-4137-48bc-a9f3-7cf5c8819415-kube-api-access-qfzjn\") pod \"c6019750-4137-48bc-a9f3-7cf5c8819415\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") "
Oct 31 04:36:17.879394 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.879367    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-metrics-certs\") pod \"c6019750-4137-48bc-a9f3-7cf5c8819415\" (UID: \"c6019750-4137-48bc-a9f3-7cf5c8819415\") "
Oct 31 04:36:17.879629 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.879597    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6019750-4137-48bc-a9f3-7cf5c8819415-openshift-service-ca" pod "c6019750-4137-48bc-a9f3-7cf5c8819415" (UID: "c6019750-4137-48bc-a9f3-7cf5c8819415"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:36:17.879758 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.879739    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6019750-4137-48bc-a9f3-7cf5c8819415-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:17.881441 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.881411    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-maas-api-tls" pod "c6019750-4137-48bc-a9f3-7cf5c8819415" (UID: "c6019750-4137-48bc-a9f3-7cf5c8819415"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:36:17.881560 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.881451    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6019750-4137-48bc-a9f3-7cf5c8819415-kube-api-access-qfzjn" pod "c6019750-4137-48bc-a9f3-7cf5c8819415" (UID: "c6019750-4137-48bc-a9f3-7cf5c8819415"). InnerVolumeSpecName "kube-api-access-qfzjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:36:17.881560 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.881489    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-metrics-certs" pod "c6019750-4137-48bc-a9f3-7cf5c8819415" (UID: "c6019750-4137-48bc-a9f3-7cf5c8819415"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:36:17.980346 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.980308    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:17.980346 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.980340    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfzjn\" (UniqueName: \"kubernetes.io/projected/c6019750-4137-48bc-a9f3-7cf5c8819415-kube-api-access-qfzjn\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:17.980346 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:17.980350    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6019750-4137-48bc-a9f3-7cf5c8819415-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:18.605539 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:18.605502    4900 generic.go:356] "Generic (PLEG): container finished" podID="c6019750-4137-48bc-a9f3-7cf5c8819415" containerID="2731c1e27ff7a8c094ae543039809bd0ac02c705620c5b584b68ebaaf7cfcbe6" exitCode=0
Oct 31 04:36:18.605732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:18.605547    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"
Oct 31 04:36:18.605732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:18.605564    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r" event={"ID":"c6019750-4137-48bc-a9f3-7cf5c8819415","Type":"ContainerDied","Data":"2731c1e27ff7a8c094ae543039809bd0ac02c705620c5b584b68ebaaf7cfcbe6"}
Oct 31 04:36:18.605732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:18.605610    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r" event={"ID":"c6019750-4137-48bc-a9f3-7cf5c8819415","Type":"ContainerDied","Data":"26ee80aa0b02dbf3a0dc5e03a0fc98a97128ee50cec812b5a0f3143ba749aa07"}
Oct 31 04:36:18.605732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:18.605630    4900 scope.go:122] "RemoveContainer" containerID="2731c1e27ff7a8c094ae543039809bd0ac02c705620c5b584b68ebaaf7cfcbe6"
Oct 31 04:36:18.615321 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:18.615301    4900 scope.go:122] "RemoveContainer" containerID="2731c1e27ff7a8c094ae543039809bd0ac02c705620c5b584b68ebaaf7cfcbe6"
Oct 31 04:36:18.615624 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:36:18.615599    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2731c1e27ff7a8c094ae543039809bd0ac02c705620c5b584b68ebaaf7cfcbe6\": container with ID starting with 2731c1e27ff7a8c094ae543039809bd0ac02c705620c5b584b68ebaaf7cfcbe6 not found: ID does not exist" containerID="2731c1e27ff7a8c094ae543039809bd0ac02c705620c5b584b68ebaaf7cfcbe6"
Oct 31 04:36:18.615676 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:18.615634    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2731c1e27ff7a8c094ae543039809bd0ac02c705620c5b584b68ebaaf7cfcbe6"} err="failed to get container status \"2731c1e27ff7a8c094ae543039809bd0ac02c705620c5b584b68ebaaf7cfcbe6\": rpc error: code = NotFound desc = could not find container \"2731c1e27ff7a8c094ae543039809bd0ac02c705620c5b584b68ebaaf7cfcbe6\": container with ID starting with 2731c1e27ff7a8c094ae543039809bd0ac02c705620c5b584b68ebaaf7cfcbe6 not found: ID does not exist"
Oct 31 04:36:18.768236 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:18.768205    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"]
Oct 31 04:36:18.818164 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:18.818128    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-bbc981-594559f9c9-clk6r"]
Oct 31 04:36:19.151572 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:19.151542    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6019750-4137-48bc-a9f3-7cf5c8819415" path="/var/lib/kubelet/pods/c6019750-4137-48bc-a9f3-7cf5c8819415/volumes"
Oct 31 04:36:23.073004 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:23.072968    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:23.282980 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:23.282943    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:25.474186 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.474145    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s" podUID="e57a668a-c6a5-492d-b2f5-c9e8693fb054" containerName="main" containerID="cri-o://127094a38e65da2b992fd7e1e1aecb327c54dddded118f36327158859664661a" gracePeriod=45
Oct 31 04:36:25.649339 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.649304    4900 generic.go:356] "Generic (PLEG): container finished" podID="e57a668a-c6a5-492d-b2f5-c9e8693fb054" containerID="127094a38e65da2b992fd7e1e1aecb327c54dddded118f36327158859664661a" exitCode=0
Oct 31 04:36:25.649489 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.649385    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s" event={"ID":"e57a668a-c6a5-492d-b2f5-c9e8693fb054","Type":"ContainerDied","Data":"127094a38e65da2b992fd7e1e1aecb327c54dddded118f36327158859664661a"}
Oct 31 04:36:25.725102 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.725075    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:36:25.758247 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.758210    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-home\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-home\") pod \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") "
Oct 31 04:36:25.758415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.758264    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57a668a-c6a5-492d-b2f5-c9e8693fb054-kube-api-access-d687v\" (UniqueName: \"kubernetes.io/projected/e57a668a-c6a5-492d-b2f5-c9e8693fb054-kube-api-access-d687v\") pod \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") "
Oct 31 04:36:25.758415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.758372    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e57a668a-c6a5-492d-b2f5-c9e8693fb054-tls-certs\" (UniqueName: \"kubernetes.io/secret/e57a668a-c6a5-492d-b2f5-c9e8693fb054-tls-certs\") pod \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") "
Oct 31 04:36:25.758415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.758394    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-model-cache\") pod \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") "
Oct 31 04:36:25.758600 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.758419    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-tmp-dir\") pod \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") "
Oct 31 04:36:25.758600 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.758470    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-dshm\") pod \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\" (UID: \"e57a668a-c6a5-492d-b2f5-c9e8693fb054\") "
Oct 31 04:36:25.758600 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.758501    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-home" pod "e57a668a-c6a5-492d-b2f5-c9e8693fb054" (UID: "e57a668a-c6a5-492d-b2f5-c9e8693fb054"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:36:25.758757 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.758683    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-tmp-dir" pod "e57a668a-c6a5-492d-b2f5-c9e8693fb054" (UID: "e57a668a-c6a5-492d-b2f5-c9e8693fb054"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:36:25.758757 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.758702    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-model-cache" pod "e57a668a-c6a5-492d-b2f5-c9e8693fb054" (UID: "e57a668a-c6a5-492d-b2f5-c9e8693fb054"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:36:25.758984 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.758967    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:25.759059 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.758987    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:25.759059 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.758997    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:25.760619 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.760591    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e57a668a-c6a5-492d-b2f5-c9e8693fb054-tls-certs" pod "e57a668a-c6a5-492d-b2f5-c9e8693fb054" (UID: "e57a668a-c6a5-492d-b2f5-c9e8693fb054"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:36:25.760619 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.760608    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-dshm" pod "e57a668a-c6a5-492d-b2f5-c9e8693fb054" (UID: "e57a668a-c6a5-492d-b2f5-c9e8693fb054"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:36:25.760796 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.760626    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57a668a-c6a5-492d-b2f5-c9e8693fb054-kube-api-access-d687v" pod "e57a668a-c6a5-492d-b2f5-c9e8693fb054" (UID: "e57a668a-c6a5-492d-b2f5-c9e8693fb054"). InnerVolumeSpecName "kube-api-access-d687v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:36:25.860437 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.860349    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e57a668a-c6a5-492d-b2f5-c9e8693fb054-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:25.860437 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.860394    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d687v\" (UniqueName: \"kubernetes.io/projected/e57a668a-c6a5-492d-b2f5-c9e8693fb054-kube-api-access-d687v\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:25.860437 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:25.860412    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e57a668a-c6a5-492d-b2f5-c9e8693fb054-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:26.656072 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:26.656043    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"
Oct 31 04:36:26.656495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:26.656088    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s" event={"ID":"e57a668a-c6a5-492d-b2f5-c9e8693fb054","Type":"ContainerDied","Data":"b217693323c44d20e2138900168ebe5b83753255d437a073249cadc454864768"}
Oct 31 04:36:26.656495 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:26.656141    4900 scope.go:122] "RemoveContainer" containerID="127094a38e65da2b992fd7e1e1aecb327c54dddded118f36327158859664661a"
Oct 31 04:36:26.819855 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:26.819820    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"]
Oct 31 04:36:26.869722 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:26.869681    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-bbc981/simulator-single-main-bbc981-kserve-6754b6cdd9-n5s9s"]
Oct 31 04:36:27.157899 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:27.157851    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e57a668a-c6a5-492d-b2f5-c9e8693fb054" path="/var/lib/kubelet/pods/e57a668a-c6a5-492d-b2f5-c9e8693fb054/volumes"
Oct 31 04:36:27.160044 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:27.160005    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr" podUID="f9042e89-be27-4be6-9820-b5bea3a61a9d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:36:27.160208 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:27.160019    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4" podUID="56a8662f-7bc5-42c5-811b-c309968e090b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:36:36.983407 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:36:36.983362    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:36:37.154401 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.154359    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr" podUID="f9042e89-be27-4be6-9820-b5bea3a61a9d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:36:37.154566 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.154366    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4" podUID="56a8662f-7bc5-42c5-811b-c309968e090b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:36:37.713527 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.713495    4900 generic.go:356] "Generic (PLEG): container finished" podID="f9042e89-be27-4be6-9820-b5bea3a61a9d" containerID="f13e2c48ecd14799e05097f6b0e1ece8588b6366a992d0f7973a680804f0513a" exitCode=0
Oct 31 04:36:37.713708 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.713566    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr" event={"ID":"f9042e89-be27-4be6-9820-b5bea3a61a9d","Type":"ContainerDied","Data":"f13e2c48ecd14799e05097f6b0e1ece8588b6366a992d0f7973a680804f0513a"}
Oct 31 04:36:37.713708 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.713623    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr" event={"ID":"f9042e89-be27-4be6-9820-b5bea3a61a9d","Type":"ContainerDied","Data":"43d00c71813fa563f6652c36263d2ae313b1a63f233e418166972d0ecda49438"}
Oct 31 04:36:37.713708 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.713648    4900 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43d00c71813fa563f6652c36263d2ae313b1a63f233e418166972d0ecda49438"
Oct 31 04:36:37.715204 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.715173    4900 generic.go:356] "Generic (PLEG): container finished" podID="56a8662f-7bc5-42c5-811b-c309968e090b" containerID="356105836e4ed892e38714f211b43e14b5c1332f9a351895aaf2a9b32eac0bcb" exitCode=0
Oct 31 04:36:37.715319 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.715236    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4" event={"ID":"56a8662f-7bc5-42c5-811b-c309968e090b","Type":"ContainerDied","Data":"356105836e4ed892e38714f211b43e14b5c1332f9a351895aaf2a9b32eac0bcb"}
Oct 31 04:36:37.717020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.716993    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"
Oct 31 04:36:37.788576 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.788536    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9042e89-be27-4be6-9820-b5bea3a61a9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9042e89-be27-4be6-9820-b5bea3a61a9d-plugins-config-volume\") pod \"f9042e89-be27-4be6-9820-b5bea3a61a9d\" (UID: \"f9042e89-be27-4be6-9820-b5bea3a61a9d\") "
Oct 31 04:36:37.788871 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.788606    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9042e89-be27-4be6-9820-b5bea3a61a9d-kube-api-access-wk5df\" (UniqueName: \"kubernetes.io/projected/f9042e89-be27-4be6-9820-b5bea3a61a9d-kube-api-access-wk5df\") pod \"f9042e89-be27-4be6-9820-b5bea3a61a9d\" (UID: \"f9042e89-be27-4be6-9820-b5bea3a61a9d\") "
Oct 31 04:36:37.789039 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.789010    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9042e89-be27-4be6-9820-b5bea3a61a9d-plugins-config-volume" pod "f9042e89-be27-4be6-9820-b5bea3a61a9d" (UID: "f9042e89-be27-4be6-9820-b5bea3a61a9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:36:37.789176 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.789152    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9042e89-be27-4be6-9820-b5bea3a61a9d-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:37.792088 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.792061    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9042e89-be27-4be6-9820-b5bea3a61a9d-kube-api-access-wk5df" pod "f9042e89-be27-4be6-9820-b5bea3a61a9d" (UID: "f9042e89-be27-4be6-9820-b5bea3a61a9d"). InnerVolumeSpecName "kube-api-access-wk5df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:36:37.816241 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.816216    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"
Oct 31 04:36:37.889955 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.889922    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a8662f-7bc5-42c5-811b-c309968e090b-kube-api-access-z7pdm\" (UniqueName: \"kubernetes.io/projected/56a8662f-7bc5-42c5-811b-c309968e090b-kube-api-access-z7pdm\") pod \"56a8662f-7bc5-42c5-811b-c309968e090b\" (UID: \"56a8662f-7bc5-42c5-811b-c309968e090b\") "
Oct 31 04:36:37.889955 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.889961    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a8662f-7bc5-42c5-811b-c309968e090b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a8662f-7bc5-42c5-811b-c309968e090b-plugins-config-volume\") pod \"56a8662f-7bc5-42c5-811b-c309968e090b\" (UID: \"56a8662f-7bc5-42c5-811b-c309968e090b\") "
Oct 31 04:36:37.890263 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.890248    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk5df\" (UniqueName: \"kubernetes.io/projected/f9042e89-be27-4be6-9820-b5bea3a61a9d-kube-api-access-wk5df\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:37.890376 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.890353    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a8662f-7bc5-42c5-811b-c309968e090b-plugins-config-volume" pod "56a8662f-7bc5-42c5-811b-c309968e090b" (UID: "56a8662f-7bc5-42c5-811b-c309968e090b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:36:37.892173 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.892151    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a8662f-7bc5-42c5-811b-c309968e090b-kube-api-access-z7pdm" pod "56a8662f-7bc5-42c5-811b-c309968e090b" (UID: "56a8662f-7bc5-42c5-811b-c309968e090b"). InnerVolumeSpecName "kube-api-access-z7pdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:36:37.991329 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.991298    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7pdm\" (UniqueName: \"kubernetes.io/projected/56a8662f-7bc5-42c5-811b-c309968e090b-kube-api-access-z7pdm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:37.991329 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:37.991327    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56a8662f-7bc5-42c5-811b-c309968e090b-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:38.720305 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:38.720261    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4" event={"ID":"56a8662f-7bc5-42c5-811b-c309968e090b","Type":"ContainerDied","Data":"579e1051d65d9fbc86b99f7153b75a1b5fe6e73138c725077f126fdee7965391"}
Oct 31 04:36:38.720498 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:38.720327    4900 scope.go:122] "RemoveContainer" containerID="356105836e4ed892e38714f211b43e14b5c1332f9a351895aaf2a9b32eac0bcb"
Oct 31 04:36:38.720498 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:38.720279    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"
Oct 31 04:36:38.720498 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:38.720279    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"
Oct 31 04:36:38.883209 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:38.883170    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"]
Oct 31 04:36:38.933091 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:38.933060    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bbc981-7fd54db94ftpxr"]
Oct 31 04:36:39.042160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:39.042127    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"]
Oct 31 04:36:39.044001 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:39.043974    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bbc981-59b4bqbkl4"]
Oct 31 04:36:39.151526 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:39.151445    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a8662f-7bc5-42c5-811b-c309968e090b" path="/var/lib/kubelet/pods/56a8662f-7bc5-42c5-811b-c309968e090b/volumes"
Oct 31 04:36:39.151875 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:39.151861    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9042e89-be27-4be6-9820-b5bea3a61a9d" path="/var/lib/kubelet/pods/f9042e89-be27-4be6-9820-b5bea3a61a9d/volumes"
Oct 31 04:36:47.098089 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:47.098059    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:47.518595 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:47.518558    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:48.436812 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:48.436734    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:48.667262 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:48.667222    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:50.268218 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:50.268180    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:50.628648 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:50.628545    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:51.166068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:51.166035    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"]
Oct 31 04:36:51.166261 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:51.166240    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc" podUID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" containerName="istio-proxy" containerID="cri-o://51ffeba5427586589204dd442e94a88abe319a3f883b3cf3288c6304a3d18c4b" gracePeriod=30
Oct 31 04:36:51.247239 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:51.247208    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:53.303213 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:53.303178    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:53.512222 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:53.512189    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:36:56.416428 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.416404    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:36:56.478905 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.478821    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-token\" (UniqueName: \"kubernetes.io/projected/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-token\") pod \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") "
Oct 31 04:36:56.478905 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.478864    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-envoy\") pod \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") "
Oct 31 04:36:56.478905 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.478897    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-data\") pod \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") "
Oct 31 04:36:56.479152 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479024    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-workload-socket\") pod \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") "
Oct 31 04:36:56.479152 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479072    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-kube-api-access-xwn5l\" (UniqueName: \"kubernetes.io/projected/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-kube-api-access-xwn5l\") pod \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") "
Oct 31 04:36:56.479152 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479099    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-podinfo\") pod \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") "
Oct 31 04:36:56.479152 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479139    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-credential-socket\") pod \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") "
Oct 31 04:36:56.479342 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479170    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-data" pod "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" (UID: "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:36:56.479342 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479201    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-workload-certs\") pod \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") "
Oct 31 04:36:56.479342 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479322    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istiod-ca-cert\") pod \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\" (UID: \"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993\") "
Oct 31 04:36:56.479525 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479342    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-workload-socket" pod "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" (UID: "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:36:56.479584 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479504    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-credential-socket" pod "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" (UID: "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:36:56.479584 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479574    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-workload-certs" pod "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" (UID: "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:36:56.479761 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479742    4900 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-workload-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:56.479830 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479769    4900 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-credential-socket\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:56.479830 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479808    4900 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-workload-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:56.479830 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479825    4900 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-data\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:56.479950 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.479857    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istiod-ca-cert" pod "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" (UID: "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:36:56.481220 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.481192    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-kube-api-access-xwn5l" pod "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" (UID: "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993"). InnerVolumeSpecName "kube-api-access-xwn5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:36:56.481220 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.481203    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-token" pod "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" (UID: "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:36:56.481501 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.481474    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-envoy" pod "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" (UID: "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:36:56.481650 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.481628    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-podinfo" pod "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" (UID: "bb90bf4f-d30f-4af7-8bdf-355a2a3b0993"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:36:56.581238 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.581206    4900 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istiod-ca-cert\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:56.581238 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.581235    4900 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-token\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:56.581238 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.581246    4900 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-envoy\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:56.581488 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.581257    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwn5l\" (UniqueName: \"kubernetes.io/projected/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-kube-api-access-xwn5l\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:56.581488 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.581267    4900 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993-istio-podinfo\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:36:56.806733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.806694    4900 generic.go:356] "Generic (PLEG): container finished" podID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" containerID="51ffeba5427586589204dd442e94a88abe319a3f883b3cf3288c6304a3d18c4b" exitCode=0
Oct 31 04:36:56.806944 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.806753    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"
Oct 31 04:36:56.806944 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.806775    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc" event={"ID":"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993","Type":"ContainerDied","Data":"51ffeba5427586589204dd442e94a88abe319a3f883b3cf3288c6304a3d18c4b"}
Oct 31 04:36:56.806944 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.806853    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc" event={"ID":"bb90bf4f-d30f-4af7-8bdf-355a2a3b0993","Type":"ContainerDied","Data":"0f6bc945493efd16d3d33eb673c73545ba185d42b73b3af2734ec78c2fd37318"}
Oct 31 04:36:56.806944 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.806876    4900 scope.go:122] "RemoveContainer" containerID="51ffeba5427586589204dd442e94a88abe319a3f883b3cf3288c6304a3d18c4b"
Oct 31 04:36:56.815725 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.815702    4900 scope.go:122] "RemoveContainer" containerID="d0f018d4aae2604ad64c176f5348a173be1773621d5bceba42757daf1ebee6b9"
Oct 31 04:36:56.823593 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.823574    4900 scope.go:122] "RemoveContainer" containerID="51ffeba5427586589204dd442e94a88abe319a3f883b3cf3288c6304a3d18c4b"
Oct 31 04:36:56.823881 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:36:56.823860    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ffeba5427586589204dd442e94a88abe319a3f883b3cf3288c6304a3d18c4b\": container with ID starting with 51ffeba5427586589204dd442e94a88abe319a3f883b3cf3288c6304a3d18c4b not found: ID does not exist" containerID="51ffeba5427586589204dd442e94a88abe319a3f883b3cf3288c6304a3d18c4b"
Oct 31 04:36:56.823960 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.823889    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ffeba5427586589204dd442e94a88abe319a3f883b3cf3288c6304a3d18c4b"} err="failed to get container status \"51ffeba5427586589204dd442e94a88abe319a3f883b3cf3288c6304a3d18c4b\": rpc error: code = NotFound desc = could not find container \"51ffeba5427586589204dd442e94a88abe319a3f883b3cf3288c6304a3d18c4b\": container with ID starting with 51ffeba5427586589204dd442e94a88abe319a3f883b3cf3288c6304a3d18c4b not found: ID does not exist"
Oct 31 04:36:56.823960 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.823909    4900 scope.go:122] "RemoveContainer" containerID="d0f018d4aae2604ad64c176f5348a173be1773621d5bceba42757daf1ebee6b9"
Oct 31 04:36:56.824129 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:36:56.824115    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f018d4aae2604ad64c176f5348a173be1773621d5bceba42757daf1ebee6b9\": container with ID starting with d0f018d4aae2604ad64c176f5348a173be1773621d5bceba42757daf1ebee6b9 not found: ID does not exist" containerID="d0f018d4aae2604ad64c176f5348a173be1773621d5bceba42757daf1ebee6b9"
Oct 31 04:36:56.824166 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.824132    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f018d4aae2604ad64c176f5348a173be1773621d5bceba42757daf1ebee6b9"} err="failed to get container status \"d0f018d4aae2604ad64c176f5348a173be1773621d5bceba42757daf1ebee6b9\": rpc error: code = NotFound desc = could not find container \"d0f018d4aae2604ad64c176f5348a173be1773621d5bceba42757daf1ebee6b9\": container with ID starting with d0f018d4aae2604ad64c176f5348a173be1773621d5bceba42757daf1ebee6b9 not found: ID does not exist"
Oct 31 04:36:56.971138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.971101    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"]
Oct 31 04:36:56.976801 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:56.976771    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bbc981-openshift-default-7545c598f76d2dc"]
Oct 31 04:36:57.152766 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:36:57.152685    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" path="/var/lib/kubelet/pods/bb90bf4f-d30f-4af7-8bdf-355a2a3b0993/volumes"
Oct 31 04:37:06.070334 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070299    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"]
Oct 31 04:37:06.070733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070691    4900 cpu_manager.go:414] "containerMap: removing container" podUID="f9042e89-be27-4be6-9820-b5bea3a61a9d" containerName="payload-processing"
Oct 31 04:37:06.070733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070706    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9042e89-be27-4be6-9820-b5bea3a61a9d" containerName="payload-processing"
Oct 31 04:37:06.070733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070725    4900 cpu_manager.go:414] "containerMap: removing container" podUID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" containerName="istio-proxy"
Oct 31 04:37:06.070733 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070731    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" containerName="istio-proxy"
Oct 31 04:37:06.070904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070745    4900 cpu_manager.go:414] "containerMap: removing container" podUID="e57a668a-c6a5-492d-b2f5-c9e8693fb054" containerName="main"
Oct 31 04:37:06.070904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070751    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57a668a-c6a5-492d-b2f5-c9e8693fb054" containerName="main"
Oct 31 04:37:06.070904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070762    4900 cpu_manager.go:414] "containerMap: removing container" podUID="c6019750-4137-48bc-a9f3-7cf5c8819415" containerName="maas-api"
Oct 31 04:37:06.070904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070768    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6019750-4137-48bc-a9f3-7cf5c8819415" containerName="maas-api"
Oct 31 04:37:06.070904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070775    4900 cpu_manager.go:414] "containerMap: removing container" podUID="56a8662f-7bc5-42c5-811b-c309968e090b" containerName="payload-pre-processing"
Oct 31 04:37:06.070904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070779    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a8662f-7bc5-42c5-811b-c309968e090b" containerName="payload-pre-processing"
Oct 31 04:37:06.070904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070806    4900 cpu_manager.go:414] "containerMap: removing container" podUID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" containerName="istio-proxy"
Oct 31 04:37:06.070904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070815    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" containerName="istio-proxy"
Oct 31 04:37:06.070904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070881    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9042e89-be27-4be6-9820-b5bea3a61a9d" containerName="payload-processing"
Oct 31 04:37:06.070904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070893    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" containerName="istio-proxy"
Oct 31 04:37:06.070904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070901    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57a668a-c6a5-492d-b2f5-c9e8693fb054" containerName="main"
Oct 31 04:37:06.070904 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070907    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6019750-4137-48bc-a9f3-7cf5c8819415" containerName="maas-api"
Oct 31 04:37:06.071263 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070913    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a8662f-7bc5-42c5-811b-c309968e090b" containerName="payload-pre-processing"
Oct 31 04:37:06.071263 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.070920    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb90bf4f-d30f-4af7-8bdf-355a2a3b0993" containerName="istio-proxy"
Oct 31 04:37:06.080053 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.080025    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.127962 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.127926    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"]
Oct 31 04:37:06.176299 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.176271    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-12997f0c-kserve-self-signed-certs\""
Oct 31 04:37:06.177740 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.177709    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-model-cache\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.177889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.177757    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-dshm\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.177968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.177933    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-tmp-dir\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.178032 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.177975    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73cf5c8-89c6-4667-823c-de169e09ab83-tls-certs\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.178310 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.178282    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-home\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.178405 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.178349    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gx7z\" (UniqueName: \"kubernetes.io/projected/e73cf5c8-89c6-4667-823c-de169e09ab83-kube-api-access-8gx7z\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.237182 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.237147    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:06.279652 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.279621    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-model-cache\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.279652 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.279658    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-dshm\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.279888 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.279783    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-tmp-dir\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.279888 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.279840    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73cf5c8-89c6-4667-823c-de169e09ab83-tls-certs\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.279888 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.279869    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-home\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.280070 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.279893    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gx7z\" (UniqueName: \"kubernetes.io/projected/e73cf5c8-89c6-4667-823c-de169e09ab83-kube-api-access-8gx7z\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.280134 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.280113    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-model-cache\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.280197 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.280145    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-tmp-dir\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.280197 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.280170    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-home\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.282020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.282001    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-dshm\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.282316 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.282294    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73cf5c8-89c6-4667-823c-de169e09ab83-tls-certs\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.337477 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.337418    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gx7z\" (UniqueName: \"kubernetes.io/projected/e73cf5c8-89c6-4667-823c-de169e09ab83-kube-api-access-8gx7z\") pod \"e2e-same-id-a-12997f0c-kserve-6686955949-lf42k\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") " pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.393627 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.393603    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:06.762585 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:37:06.762547    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73cf5c8_89c6_4667_823c_de169e09ab83.slice/crio-25078d08717414f45dbb06e9cfa0c312af46192d52703283a81e611d02fa79a2 WatchSource:0}: Error finding container 25078d08717414f45dbb06e9cfa0c312af46192d52703283a81e611d02fa79a2: Status 404 returned error can't find the container with id 25078d08717414f45dbb06e9cfa0c312af46192d52703283a81e611d02fa79a2
Oct 31 04:37:06.764036 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.764010    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"]
Oct 31 04:37:06.855750 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.855720    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k" event={"ID":"e73cf5c8-89c6-4667-823c-de169e09ab83","Type":"ContainerStarted","Data":"183e7de8d096df379e3a214b51f0183185bb48ba2c64239addf0df535263a0f2"}
Oct 31 04:37:06.855938 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.855760    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k" event={"ID":"e73cf5c8-89c6-4667-823c-de169e09ab83","Type":"ContainerStarted","Data":"25078d08717414f45dbb06e9cfa0c312af46192d52703283a81e611d02fa79a2"}
Oct 31 04:37:06.970652 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:06.970592    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k" podStartSLOduration=0.970575105 podStartE2EDuration="970.575105ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:37:06.969832579 +0000 UTC m=+5379.936165924" watchObservedRunningTime="2026-10-31 04:37:06.970575105 +0000 UTC m=+5379.936908453"
Oct 31 04:37:06.985088 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:37:06.985058    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:37:07.394540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:07.394512    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:10.364761 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:10.364721    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:10.568954 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:10.568920    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:12.915369 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:12.915322    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:16.394738 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:16.394704    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:16.407246 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:16.407216    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:16.928622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:16.928588    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:37:29.046263 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:29.046233    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:37:29.046771 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:29.046687    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:37:30.290875 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.290843    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"]
Oct 31 04:37:30.295846 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.295825    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.345630 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.345594    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-tls-certs\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.345856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.345730    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-model-cache\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.345856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.345810    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-dshm\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.345856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.345846    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5plx\" (UniqueName: \"kubernetes.io/projected/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-kube-api-access-b5plx\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.346090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.345890    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-tmp-dir\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.346322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.345957    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-home\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.347641 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.347619    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"]
Oct 31 04:37:30.395617 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.395583    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-12997f0c-kserve-self-signed-certs\""
Oct 31 04:37:30.447030 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.446991    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-tmp-dir\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.447227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.447077    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-home\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.447227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.447140    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-tls-certs\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.447227 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.447222    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-model-cache\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.447451 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.447273    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-dshm\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.447451 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.447306    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5plx\" (UniqueName: \"kubernetes.io/projected/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-kube-api-access-b5plx\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.448034 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.447999    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-tmp-dir\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.448259 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.448242    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-home\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.448921 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.448868    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-model-cache\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.451283 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.451258    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-tls-certs\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.451632 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.451609    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-dshm\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.458661 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.458632    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:37:30.504026 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.503989    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5plx\" (UniqueName: \"kubernetes.io/projected/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-kube-api-access-b5plx\") pod \"e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") " pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.607417 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.607322    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:30.977052 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:37:30.977019    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c34f07b_cec0_42c5_a9d2_bc4be10fb3d4.slice/crio-f7913aa5a72aa98d9012c6552ae48cf35c298d9d9d5bc000a52298696ffd24cf WatchSource:0}: Error finding container f7913aa5a72aa98d9012c6552ae48cf35c298d9d9d5bc000a52298696ffd24cf: Status 404 returned error can't find the container with id f7913aa5a72aa98d9012c6552ae48cf35c298d9d9d5bc000a52298696ffd24cf
Oct 31 04:37:30.979037 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:30.979015    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"]
Oct 31 04:37:31.986482 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:31.986437    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768" event={"ID":"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4","Type":"ContainerStarted","Data":"8b77a8ded791b211f6019aa567c17fe13e266c29afb3dea5a551bb77af020d0c"}
Oct 31 04:37:31.986929 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:31.986490    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768" event={"ID":"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4","Type":"ContainerStarted","Data":"f7913aa5a72aa98d9012c6552ae48cf35c298d9d9d5bc000a52298696ffd24cf"}
Oct 31 04:37:32.101163 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:32.101112    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768" podStartSLOduration=2.101098028 podStartE2EDuration="2.101098028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:37:32.100319711 +0000 UTC m=+5405.066653056" watchObservedRunningTime="2026-10-31 04:37:32.101098028 +0000 UTC m=+5405.067431372"
Oct 31 04:37:32.607760 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:32.607727    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:36.983370 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:37:36.983332    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:37:40.608200 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:40.608099    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:40.620723 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:40.620699    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:37:41.047007 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:37:41.046977    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:38:06.985134 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:38:06.985098    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:38:12.041714 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:12.041676    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"]
Oct 31 04:38:12.179891 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:12.179851    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:13.512566 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:13.512499    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:14.609888 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:14.609857    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"]
Oct 31 04:38:14.685848 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:14.685822    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:16.153408 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:16.153372    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:16.373162 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:16.373132    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:38:27.066564 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.066520    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k" podUID="e73cf5c8-89c6-4667-823c-de169e09ab83" containerName="main" containerID="cri-o://183e7de8d096df379e3a214b51f0183185bb48ba2c64239addf0df535263a0f2" gracePeriod=45
Oct 31 04:38:27.316537 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.316514    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:38:27.340678 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.340618    4900 generic.go:356] "Generic (PLEG): container finished" podID="e73cf5c8-89c6-4667-823c-de169e09ab83" containerID="183e7de8d096df379e3a214b51f0183185bb48ba2c64239addf0df535263a0f2" exitCode=0
Oct 31 04:38:27.340678 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.340660    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k" event={"ID":"e73cf5c8-89c6-4667-823c-de169e09ab83","Type":"ContainerDied","Data":"183e7de8d096df379e3a214b51f0183185bb48ba2c64239addf0df535263a0f2"}
Oct 31 04:38:27.340678 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.340665    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"
Oct 31 04:38:27.340911 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.340694    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k" event={"ID":"e73cf5c8-89c6-4667-823c-de169e09ab83","Type":"ContainerDied","Data":"25078d08717414f45dbb06e9cfa0c312af46192d52703283a81e611d02fa79a2"}
Oct 31 04:38:27.340911 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.340711    4900 scope.go:122] "RemoveContainer" containerID="183e7de8d096df379e3a214b51f0183185bb48ba2c64239addf0df535263a0f2"
Oct 31 04:38:27.348728 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.348712    4900 scope.go:122] "RemoveContainer" containerID="183e7de8d096df379e3a214b51f0183185bb48ba2c64239addf0df535263a0f2"
Oct 31 04:38:27.349009 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:38:27.348990    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"183e7de8d096df379e3a214b51f0183185bb48ba2c64239addf0df535263a0f2\": container with ID starting with 183e7de8d096df379e3a214b51f0183185bb48ba2c64239addf0df535263a0f2 not found: ID does not exist" containerID="183e7de8d096df379e3a214b51f0183185bb48ba2c64239addf0df535263a0f2"
Oct 31 04:38:27.349091 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.349022    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"183e7de8d096df379e3a214b51f0183185bb48ba2c64239addf0df535263a0f2"} err="failed to get container status \"183e7de8d096df379e3a214b51f0183185bb48ba2c64239addf0df535263a0f2\": rpc error: code = NotFound desc = could not find container \"183e7de8d096df379e3a214b51f0183185bb48ba2c64239addf0df535263a0f2\": container with ID starting with 183e7de8d096df379e3a214b51f0183185bb48ba2c64239addf0df535263a0f2 not found: ID does not exist"
Oct 31 04:38:27.369122 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.369099    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e73cf5c8-89c6-4667-823c-de169e09ab83-tls-certs\" (UniqueName: \"kubernetes.io/secret/e73cf5c8-89c6-4667-823c-de169e09ab83-tls-certs\") pod \"e73cf5c8-89c6-4667-823c-de169e09ab83\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") "
Oct 31 04:38:27.369209 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.369194    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-tmp-dir\") pod \"e73cf5c8-89c6-4667-823c-de169e09ab83\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") "
Oct 31 04:38:27.369255 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.369218    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-dshm\") pod \"e73cf5c8-89c6-4667-823c-de169e09ab83\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") "
Oct 31 04:38:27.369255 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.369243    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73cf5c8-89c6-4667-823c-de169e09ab83-kube-api-access-8gx7z\" (UniqueName: \"kubernetes.io/projected/e73cf5c8-89c6-4667-823c-de169e09ab83-kube-api-access-8gx7z\") pod \"e73cf5c8-89c6-4667-823c-de169e09ab83\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") "
Oct 31 04:38:27.369374 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.369280    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-home\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-home\") pod \"e73cf5c8-89c6-4667-823c-de169e09ab83\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") "
Oct 31 04:38:27.369437 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.369385    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-model-cache\") pod \"e73cf5c8-89c6-4667-823c-de169e09ab83\" (UID: \"e73cf5c8-89c6-4667-823c-de169e09ab83\") "
Oct 31 04:38:27.369533 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.369513    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-tmp-dir" pod "e73cf5c8-89c6-4667-823c-de169e09ab83" (UID: "e73cf5c8-89c6-4667-823c-de169e09ab83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:27.369533 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.369524    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-home" pod "e73cf5c8-89c6-4667-823c-de169e09ab83" (UID: "e73cf5c8-89c6-4667-823c-de169e09ab83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:27.369682 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.369660    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-model-cache" pod "e73cf5c8-89c6-4667-823c-de169e09ab83" (UID: "e73cf5c8-89c6-4667-823c-de169e09ab83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:27.369844 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.369822    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:27.369844 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.369844    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:27.370041 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.369858    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:27.371226 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.371199    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e73cf5c8-89c6-4667-823c-de169e09ab83-tls-certs" pod "e73cf5c8-89c6-4667-823c-de169e09ab83" (UID: "e73cf5c8-89c6-4667-823c-de169e09ab83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:38:27.371226 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.371217    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-dshm" pod "e73cf5c8-89c6-4667-823c-de169e09ab83" (UID: "e73cf5c8-89c6-4667-823c-de169e09ab83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:27.371346 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.371239    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73cf5c8-89c6-4667-823c-de169e09ab83-kube-api-access-8gx7z" pod "e73cf5c8-89c6-4667-823c-de169e09ab83" (UID: "e73cf5c8-89c6-4667-823c-de169e09ab83"). InnerVolumeSpecName "kube-api-access-8gx7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:38:27.470630 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.470601    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73cf5c8-89c6-4667-823c-de169e09ab83-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:27.470630 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.470628    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gx7z\" (UniqueName: \"kubernetes.io/projected/e73cf5c8-89c6-4667-823c-de169e09ab83-kube-api-access-8gx7z\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:27.470842 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.470639    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73cf5c8-89c6-4667-823c-de169e09ab83-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:27.760061 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.760032    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"]
Oct 31 04:38:27.763889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:27.763870    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-12997f0c-kserve-6686955949-lf42k"]
Oct 31 04:38:29.156803 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.156753    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73cf5c8-89c6-4667-823c-de169e09ab83" path="/var/lib/kubelet/pods/e73cf5c8-89c6-4667-823c-de169e09ab83/volumes"
Oct 31 04:38:29.629356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.629312    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768" podUID="7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4" containerName="main" containerID="cri-o://8b77a8ded791b211f6019aa567c17fe13e266c29afb3dea5a551bb77af020d0c" gracePeriod=45
Oct 31 04:38:29.879768 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.879712    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:38:29.997839 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.997807    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-tls-certs\") pod \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") "
Oct 31 04:38:29.998007 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.997852    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-kube-api-access-b5plx\" (UniqueName: \"kubernetes.io/projected/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-kube-api-access-b5plx\") pod \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") "
Oct 31 04:38:29.998007 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.997950    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-model-cache\") pod \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") "
Oct 31 04:38:29.998007 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.997988    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-home\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-home\") pod \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") "
Oct 31 04:38:29.998200 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.998014    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-tmp-dir\") pod \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") "
Oct 31 04:38:29.998200 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.998066    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-dshm\") pod \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\" (UID: \"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4\") "
Oct 31 04:38:29.998352 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.998306    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-home" pod "7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4" (UID: "7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:29.998412 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.998338    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-model-cache" pod "7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4" (UID: "7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:29.998412 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.998367    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-tmp-dir" pod "7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4" (UID: "7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:29.998780 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.998757    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:29.998920 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.998808    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:29.998920 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:29.998824    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:30.000182 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.000155    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-kube-api-access-b5plx" pod "7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4" (UID: "7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4"). InnerVolumeSpecName "kube-api-access-b5plx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:38:30.000293 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.000206    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-tls-certs" pod "7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4" (UID: "7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:38:30.000422 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.000399    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-dshm" pod "7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4" (UID: "7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:38:30.100121 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.100083    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:30.100121 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.100112    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:30.100121 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.100124    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5plx\" (UniqueName: \"kubernetes.io/projected/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4-kube-api-access-b5plx\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:38:30.356319 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.356280    4900 generic.go:356] "Generic (PLEG): container finished" podID="7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4" containerID="8b77a8ded791b211f6019aa567c17fe13e266c29afb3dea5a551bb77af020d0c" exitCode=0
Oct 31 04:38:30.356319 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.356320    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"
Oct 31 04:38:30.356872 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.356334    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768" event={"ID":"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4","Type":"ContainerDied","Data":"8b77a8ded791b211f6019aa567c17fe13e266c29afb3dea5a551bb77af020d0c"}
Oct 31 04:38:30.356872 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.356366    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768" event={"ID":"7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4","Type":"ContainerDied","Data":"f7913aa5a72aa98d9012c6552ae48cf35c298d9d9d5bc000a52298696ffd24cf"}
Oct 31 04:38:30.356872 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.356386    4900 scope.go:122] "RemoveContainer" containerID="8b77a8ded791b211f6019aa567c17fe13e266c29afb3dea5a551bb77af020d0c"
Oct 31 04:38:30.365203 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.365173    4900 scope.go:122] "RemoveContainer" containerID="8b77a8ded791b211f6019aa567c17fe13e266c29afb3dea5a551bb77af020d0c"
Oct 31 04:38:30.365475 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:38:30.365454    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b77a8ded791b211f6019aa567c17fe13e266c29afb3dea5a551bb77af020d0c\": container with ID starting with 8b77a8ded791b211f6019aa567c17fe13e266c29afb3dea5a551bb77af020d0c not found: ID does not exist" containerID="8b77a8ded791b211f6019aa567c17fe13e266c29afb3dea5a551bb77af020d0c"
Oct 31 04:38:30.365543 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.365483    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b77a8ded791b211f6019aa567c17fe13e266c29afb3dea5a551bb77af020d0c"} err="failed to get container status \"8b77a8ded791b211f6019aa567c17fe13e266c29afb3dea5a551bb77af020d0c\": rpc error: code = NotFound desc = could not find container \"8b77a8ded791b211f6019aa567c17fe13e266c29afb3dea5a551bb77af020d0c\": container with ID starting with 8b77a8ded791b211f6019aa567c17fe13e266c29afb3dea5a551bb77af020d0c not found: ID does not exist"
Oct 31 04:38:30.519684 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.519645    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"]
Oct 31 04:38:30.522015 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:30.521994    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-12997f0c-kserve-79fdf5d98f-28768"]
Oct 31 04:38:31.152139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:31.152099    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4" path="/var/lib/kubelet/pods/7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4/volumes"
Oct 31 04:38:36.982284 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:38:36.982246    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:38:58.774124 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:58.774087    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"]
Oct 31 04:38:58.774738 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:58.774719    4900 cpu_manager.go:414] "containerMap: removing container" podUID="7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4" containerName="main"
Oct 31 04:38:58.774814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:58.774740    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4" containerName="main"
Oct 31 04:38:58.774814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:58.774778    4900 cpu_manager.go:414] "containerMap: removing container" podUID="e73cf5c8-89c6-4667-823c-de169e09ab83" containerName="main"
Oct 31 04:38:58.774814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:58.774797    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73cf5c8-89c6-4667-823c-de169e09ab83" containerName="main"
Oct 31 04:38:58.774938 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:58.774866    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73cf5c8-89c6-4667-823c-de169e09ab83" containerName="main"
Oct 31 04:38:58.774938 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:58.774876    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c34f07b-cec0-42c5-a9d2-bc4be10fb3d4" containerName="main"
Oct 31 04:38:58.778011 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:58.777997    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"
Oct 31 04:38:58.830512 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:58.830483    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"]
Oct 31 04:38:58.912624 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:58.912589    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7257aff6-06c1-40ff-9e8a-8a79e40307a0-plugins-config-volume\") pod \"payload-pre-processing-6545dbfb5c-prvtw\" (UID: \"7257aff6-06c1-40ff-9e8a-8a79e40307a0\") " pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"
Oct 31 04:38:58.912811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:58.912773    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzrbk\" (UniqueName: \"kubernetes.io/projected/7257aff6-06c1-40ff-9e8a-8a79e40307a0-kube-api-access-bzrbk\") pod \"payload-pre-processing-6545dbfb5c-prvtw\" (UID: \"7257aff6-06c1-40ff-9e8a-8a79e40307a0\") " pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"
Oct 31 04:38:59.014192 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:59.014160    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzrbk\" (UniqueName: \"kubernetes.io/projected/7257aff6-06c1-40ff-9e8a-8a79e40307a0-kube-api-access-bzrbk\") pod \"payload-pre-processing-6545dbfb5c-prvtw\" (UID: \"7257aff6-06c1-40ff-9e8a-8a79e40307a0\") " pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"
Oct 31 04:38:59.014345 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:59.014224    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7257aff6-06c1-40ff-9e8a-8a79e40307a0-plugins-config-volume\") pod \"payload-pre-processing-6545dbfb5c-prvtw\" (UID: \"7257aff6-06c1-40ff-9e8a-8a79e40307a0\") " pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"
Oct 31 04:38:59.014870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:59.014853    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7257aff6-06c1-40ff-9e8a-8a79e40307a0-plugins-config-volume\") pod \"payload-pre-processing-6545dbfb5c-prvtw\" (UID: \"7257aff6-06c1-40ff-9e8a-8a79e40307a0\") " pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"
Oct 31 04:38:59.071565 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:59.071503    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzrbk\" (UniqueName: \"kubernetes.io/projected/7257aff6-06c1-40ff-9e8a-8a79e40307a0-kube-api-access-bzrbk\") pod \"payload-pre-processing-6545dbfb5c-prvtw\" (UID: \"7257aff6-06c1-40ff-9e8a-8a79e40307a0\") " pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"
Oct 31 04:38:59.090471 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:59.090449    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"
Oct 31 04:38:59.453904 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:38:59.453870    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7257aff6_06c1_40ff_9e8a_8a79e40307a0.slice/crio-99807a7567bca05e341965b404ec2dba5e815411be2031b68c572528c39846df WatchSource:0}: Error finding container 99807a7567bca05e341965b404ec2dba5e815411be2031b68c572528c39846df: Status 404 returned error can't find the container with id 99807a7567bca05e341965b404ec2dba5e815411be2031b68c572528c39846df
Oct 31 04:38:59.455988 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:59.455964    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"]
Oct 31 04:38:59.509357 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:38:59.509326    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw" event={"ID":"7257aff6-06c1-40ff-9e8a-8a79e40307a0","Type":"ContainerStarted","Data":"99807a7567bca05e341965b404ec2dba5e815411be2031b68c572528c39846df"}
Oct 31 04:39:00.515152 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:00.515120    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw" event={"ID":"7257aff6-06c1-40ff-9e8a-8a79e40307a0","Type":"ContainerStarted","Data":"f2b4f00a78bc440ab4152e0116859a3898b184ff60c7c065d8fb1af7aa9bf132"}
Oct 31 04:39:00.515530 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:00.515213    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"
Oct 31 04:39:00.629506 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:00.629447    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw" podStartSLOduration=2.629429741 podStartE2EDuration="2.629429741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:00.628640398 +0000 UTC m=+5493.594973742" watchObservedRunningTime="2026-10-31 04:39:00.629429741 +0000 UTC m=+5493.595763088"
Oct 31 04:39:06.983286 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:39:06.983240    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:39:10.895216 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:10.895138    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:11.524098 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:11.524066    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"
Oct 31 04:39:11.676408 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:11.676374    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"]
Oct 31 04:39:13.674361 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:13.674327    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:13.676728 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:13.676702    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-597dbd485d-sbjjw"]
Oct 31 04:39:13.680459 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:13.680444    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-597dbd485d-sbjjw"
Oct 31 04:39:13.732927 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:13.732900    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-597dbd485d-sbjjw"]
Oct 31 04:39:13.762585 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:13.762554    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8-plugins-config-volume\") pod \"payload-processing-597dbd485d-sbjjw\" (UID: \"6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8\") " pod="openshift-ingress/payload-processing-597dbd485d-sbjjw"
Oct 31 04:39:13.762747 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:13.762597    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jchl5\" (UniqueName: \"kubernetes.io/projected/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8-kube-api-access-jchl5\") pod \"payload-processing-597dbd485d-sbjjw\" (UID: \"6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8\") " pod="openshift-ingress/payload-processing-597dbd485d-sbjjw"
Oct 31 04:39:13.863873 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:13.863819    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8-plugins-config-volume\") pod \"payload-processing-597dbd485d-sbjjw\" (UID: \"6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8\") " pod="openshift-ingress/payload-processing-597dbd485d-sbjjw"
Oct 31 04:39:13.864052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:13.863902    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jchl5\" (UniqueName: \"kubernetes.io/projected/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8-kube-api-access-jchl5\") pod \"payload-processing-597dbd485d-sbjjw\" (UID: \"6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8\") " pod="openshift-ingress/payload-processing-597dbd485d-sbjjw"
Oct 31 04:39:13.864511 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:13.864490    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8-plugins-config-volume\") pod \"payload-processing-597dbd485d-sbjjw\" (UID: \"6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8\") " pod="openshift-ingress/payload-processing-597dbd485d-sbjjw"
Oct 31 04:39:13.921231 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:13.921203    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jchl5\" (UniqueName: \"kubernetes.io/projected/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8-kube-api-access-jchl5\") pod \"payload-processing-597dbd485d-sbjjw\" (UID: \"6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8\") " pod="openshift-ingress/payload-processing-597dbd485d-sbjjw"
Oct 31 04:39:13.993252 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:13.993231    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-597dbd485d-sbjjw"
Oct 31 04:39:14.359595 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:39:14.359565    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bfe5f98_7e9b_4c86_bfb7_c8d04815d9e8.slice/crio-158cfbb49dfcd64133d0049e2243a4539554cd29a312397778b55c09f3a9763b WatchSource:0}: Error finding container 158cfbb49dfcd64133d0049e2243a4539554cd29a312397778b55c09f3a9763b: Status 404 returned error can't find the container with id 158cfbb49dfcd64133d0049e2243a4539554cd29a312397778b55c09f3a9763b
Oct 31 04:39:14.360869 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:14.360845    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-597dbd485d-sbjjw"]
Oct 31 04:39:14.580147 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:14.580107    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-597dbd485d-sbjjw" event={"ID":"6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8","Type":"ContainerStarted","Data":"dea6809d102fc0579a605f7e7d033fc4405f42bfeb26e2bbe6d04b862a74aad4"}
Oct 31 04:39:14.580330 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:14.580159    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-597dbd485d-sbjjw" event={"ID":"6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8","Type":"ContainerStarted","Data":"158cfbb49dfcd64133d0049e2243a4539554cd29a312397778b55c09f3a9763b"}
Oct 31 04:39:14.580330 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:14.580205    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-597dbd485d-sbjjw"
Oct 31 04:39:14.693534 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:14.693434    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-597dbd485d-sbjjw" podStartSLOduration=1.693419952 podStartE2EDuration="1.693419952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:14.692137523 +0000 UTC m=+5507.658470867" watchObservedRunningTime="2026-10-31 04:39:14.693419952 +0000 UTC m=+5507.659753329"
Oct 31 04:39:16.677534 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:16.677484    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs" podUID="3158fd77-32bc-4a70-8937-8b58750d82f3" containerName="payload-pre-processing" containerID="cri-o://81ba4cfe4fa9008c8af963edd30123f97a291f337e2f78691c69d9a6c266a51f" gracePeriod=25
Oct 31 04:39:16.926848 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:16.926822    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"
Oct 31 04:39:16.996744 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:16.996720    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3158fd77-32bc-4a70-8937-8b58750d82f3-kube-api-access-pj2tw\" (UniqueName: \"kubernetes.io/projected/3158fd77-32bc-4a70-8937-8b58750d82f3-kube-api-access-pj2tw\") pod \"3158fd77-32bc-4a70-8937-8b58750d82f3\" (UID: \"3158fd77-32bc-4a70-8937-8b58750d82f3\") "
Oct 31 04:39:16.996902 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:16.996760    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3158fd77-32bc-4a70-8937-8b58750d82f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3158fd77-32bc-4a70-8937-8b58750d82f3-plugins-config-volume\") pod \"3158fd77-32bc-4a70-8937-8b58750d82f3\" (UID: \"3158fd77-32bc-4a70-8937-8b58750d82f3\") "
Oct 31 04:39:16.997183 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:16.997158    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3158fd77-32bc-4a70-8937-8b58750d82f3-plugins-config-volume" pod "3158fd77-32bc-4a70-8937-8b58750d82f3" (UID: "3158fd77-32bc-4a70-8937-8b58750d82f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:39:16.998745 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:16.998726    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3158fd77-32bc-4a70-8937-8b58750d82f3-kube-api-access-pj2tw" pod "3158fd77-32bc-4a70-8937-8b58750d82f3" (UID: "3158fd77-32bc-4a70-8937-8b58750d82f3"). InnerVolumeSpecName "kube-api-access-pj2tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:39:17.097515 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:17.097480    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj2tw\" (UniqueName: \"kubernetes.io/projected/3158fd77-32bc-4a70-8937-8b58750d82f3-kube-api-access-pj2tw\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:39:17.097515 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:17.097509    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3158fd77-32bc-4a70-8937-8b58750d82f3-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:39:17.595077 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:17.595044    4900 generic.go:356] "Generic (PLEG): container finished" podID="3158fd77-32bc-4a70-8937-8b58750d82f3" containerID="81ba4cfe4fa9008c8af963edd30123f97a291f337e2f78691c69d9a6c266a51f" exitCode=0
Oct 31 04:39:17.595275 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:17.595104    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"
Oct 31 04:39:17.595275 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:17.595132    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs" event={"ID":"3158fd77-32bc-4a70-8937-8b58750d82f3","Type":"ContainerDied","Data":"81ba4cfe4fa9008c8af963edd30123f97a291f337e2f78691c69d9a6c266a51f"}
Oct 31 04:39:17.595275 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:17.595213    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs" event={"ID":"3158fd77-32bc-4a70-8937-8b58750d82f3","Type":"ContainerDied","Data":"122451269aab11ba5bd050422bf05228021e5e70ce5f8acff6a7837ed3eff8b6"}
Oct 31 04:39:17.595275 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:17.595237    4900 scope.go:122] "RemoveContainer" containerID="81ba4cfe4fa9008c8af963edd30123f97a291f337e2f78691c69d9a6c266a51f"
Oct 31 04:39:17.603675 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:17.603658    4900 scope.go:122] "RemoveContainer" containerID="81ba4cfe4fa9008c8af963edd30123f97a291f337e2f78691c69d9a6c266a51f"
Oct 31 04:39:17.604068 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:39:17.604049    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ba4cfe4fa9008c8af963edd30123f97a291f337e2f78691c69d9a6c266a51f\": container with ID starting with 81ba4cfe4fa9008c8af963edd30123f97a291f337e2f78691c69d9a6c266a51f not found: ID does not exist" containerID="81ba4cfe4fa9008c8af963edd30123f97a291f337e2f78691c69d9a6c266a51f"
Oct 31 04:39:17.604121 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:17.604077    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ba4cfe4fa9008c8af963edd30123f97a291f337e2f78691c69d9a6c266a51f"} err="failed to get container status \"81ba4cfe4fa9008c8af963edd30123f97a291f337e2f78691c69d9a6c266a51f\": rpc error: code = NotFound desc = could not find container \"81ba4cfe4fa9008c8af963edd30123f97a291f337e2f78691c69d9a6c266a51f\": container with ID starting with 81ba4cfe4fa9008c8af963edd30123f97a291f337e2f78691c69d9a6c266a51f not found: ID does not exist"
Oct 31 04:39:17.759257 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:17.759227    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"]
Oct 31 04:39:17.809099 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:17.809068    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9pzbs"]
Oct 31 04:39:19.151820 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:19.151769    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3158fd77-32bc-4a70-8937-8b58750d82f3" path="/var/lib/kubelet/pods/3158fd77-32bc-4a70-8937-8b58750d82f3/volumes"
Oct 31 04:39:25.587940 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:25.587905    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-597dbd485d-sbjjw"
Oct 31 04:39:25.750188 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:25.750153    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"]
Oct 31 04:39:27.884773 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:27.884730    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-79468f7c9f-559sl"]
Oct 31 04:39:27.885366 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:27.885344    4900 cpu_manager.go:414] "containerMap: removing container" podUID="3158fd77-32bc-4a70-8937-8b58750d82f3" containerName="payload-pre-processing"
Oct 31 04:39:27.885461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:27.885369    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3158fd77-32bc-4a70-8937-8b58750d82f3" containerName="payload-pre-processing"
Oct 31 04:39:27.885527 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:27.885503    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="3158fd77-32bc-4a70-8937-8b58750d82f3" containerName="payload-pre-processing"
Oct 31 04:39:27.889819 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:27.889775    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-79468f7c9f-559sl"
Oct 31 04:39:27.941070 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:27.941037    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-79468f7c9f-559sl"]
Oct 31 04:39:28.012387 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.012352    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f54b64bf-7962-4b4c-ab0f-d3c35d5142ed-plugins-config-volume\") pod \"payload-processing-79468f7c9f-559sl\" (UID: \"f54b64bf-7962-4b4c-ab0f-d3c35d5142ed\") " pod="openshift-ingress/payload-processing-79468f7c9f-559sl"
Oct 31 04:39:28.012387 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.012391    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657kk\" (UniqueName: \"kubernetes.io/projected/f54b64bf-7962-4b4c-ab0f-d3c35d5142ed-kube-api-access-657kk\") pod \"payload-processing-79468f7c9f-559sl\" (UID: \"f54b64bf-7962-4b4c-ab0f-d3c35d5142ed\") " pod="openshift-ingress/payload-processing-79468f7c9f-559sl"
Oct 31 04:39:28.050853 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.050821    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-9cfb9b656-jtls9"]
Oct 31 04:39:28.054194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.054177    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-9cfb9b656-jtls9"
Oct 31 04:39:28.107049 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.107015    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-9cfb9b656-jtls9"]
Oct 31 04:39:28.113431 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.113402    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f54b64bf-7962-4b4c-ab0f-d3c35d5142ed-plugins-config-volume\") pod \"payload-processing-79468f7c9f-559sl\" (UID: \"f54b64bf-7962-4b4c-ab0f-d3c35d5142ed\") " pod="openshift-ingress/payload-processing-79468f7c9f-559sl"
Oct 31 04:39:28.113569 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.113439    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657kk\" (UniqueName: \"kubernetes.io/projected/f54b64bf-7962-4b4c-ab0f-d3c35d5142ed-kube-api-access-657kk\") pod \"payload-processing-79468f7c9f-559sl\" (UID: \"f54b64bf-7962-4b4c-ab0f-d3c35d5142ed\") " pod="openshift-ingress/payload-processing-79468f7c9f-559sl"
Oct 31 04:39:28.114381 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.114357    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f54b64bf-7962-4b4c-ab0f-d3c35d5142ed-plugins-config-volume\") pod \"payload-processing-79468f7c9f-559sl\" (UID: \"f54b64bf-7962-4b4c-ab0f-d3c35d5142ed\") " pod="openshift-ingress/payload-processing-79468f7c9f-559sl"
Oct 31 04:39:28.171244 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.171177    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-657kk\" (UniqueName: \"kubernetes.io/projected/f54b64bf-7962-4b4c-ab0f-d3c35d5142ed-kube-api-access-657kk\") pod \"payload-processing-79468f7c9f-559sl\" (UID: \"f54b64bf-7962-4b4c-ab0f-d3c35d5142ed\") " pod="openshift-ingress/payload-processing-79468f7c9f-559sl"
Oct 31 04:39:28.203197 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.203164    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-79468f7c9f-559sl"
Oct 31 04:39:28.214205 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.214177    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d926de8d-6439-41a4-8ad6-1c01833ed69c-plugins-config-volume\") pod \"payload-pre-processing-9cfb9b656-jtls9\" (UID: \"d926de8d-6439-41a4-8ad6-1c01833ed69c\") " pod="openshift-ingress/payload-pre-processing-9cfb9b656-jtls9"
Oct 31 04:39:28.214338 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.214317    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29k6n\" (UniqueName: \"kubernetes.io/projected/d926de8d-6439-41a4-8ad6-1c01833ed69c-kube-api-access-29k6n\") pod \"payload-pre-processing-9cfb9b656-jtls9\" (UID: \"d926de8d-6439-41a4-8ad6-1c01833ed69c\") " pod="openshift-ingress/payload-pre-processing-9cfb9b656-jtls9"
Oct 31 04:39:28.315448 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.315415    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29k6n\" (UniqueName: \"kubernetes.io/projected/d926de8d-6439-41a4-8ad6-1c01833ed69c-kube-api-access-29k6n\") pod \"payload-pre-processing-9cfb9b656-jtls9\" (UID: \"d926de8d-6439-41a4-8ad6-1c01833ed69c\") " pod="openshift-ingress/payload-pre-processing-9cfb9b656-jtls9"
Oct 31 04:39:28.315611 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.315551    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d926de8d-6439-41a4-8ad6-1c01833ed69c-plugins-config-volume\") pod \"payload-pre-processing-9cfb9b656-jtls9\" (UID: \"d926de8d-6439-41a4-8ad6-1c01833ed69c\") " pod="openshift-ingress/payload-pre-processing-9cfb9b656-jtls9"
Oct 31 04:39:28.316241 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.316223    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d926de8d-6439-41a4-8ad6-1c01833ed69c-plugins-config-volume\") pod \"payload-pre-processing-9cfb9b656-jtls9\" (UID: \"d926de8d-6439-41a4-8ad6-1c01833ed69c\") " pod="openshift-ingress/payload-pre-processing-9cfb9b656-jtls9"
Oct 31 04:39:28.372878 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.372844    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29k6n\" (UniqueName: \"kubernetes.io/projected/d926de8d-6439-41a4-8ad6-1c01833ed69c-kube-api-access-29k6n\") pod \"payload-pre-processing-9cfb9b656-jtls9\" (UID: \"d926de8d-6439-41a4-8ad6-1c01833ed69c\") " pod="openshift-ingress/payload-pre-processing-9cfb9b656-jtls9"
Oct 31 04:39:28.569171 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:39:28.569139    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54b64bf_7962_4b4c_ab0f_d3c35d5142ed.slice/crio-96e124e47ae862ae837595d88752d0b9730b47151af90a3a88110e437261a730 WatchSource:0}: Error finding container 96e124e47ae862ae837595d88752d0b9730b47151af90a3a88110e437261a730: Status 404 returned error can't find the container with id 96e124e47ae862ae837595d88752d0b9730b47151af90a3a88110e437261a730
Oct 31 04:39:28.570910 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.570881    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-79468f7c9f-559sl"]
Oct 31 04:39:28.647600 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.647560    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-79468f7c9f-559sl" event={"ID":"f54b64bf-7962-4b4c-ab0f-d3c35d5142ed","Type":"ContainerStarted","Data":"3b1e514a6745f6328af045a17c04ac5818847e91d12a6f10a9d8a9b910414ec5"}
Oct 31 04:39:28.647768 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.647650    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-79468f7c9f-559sl" event={"ID":"f54b64bf-7962-4b4c-ab0f-d3c35d5142ed","Type":"ContainerStarted","Data":"96e124e47ae862ae837595d88752d0b9730b47151af90a3a88110e437261a730"}
Oct 31 04:39:28.647768 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.647680    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-79468f7c9f-559sl"
Oct 31 04:39:28.667276 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.667250    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-9cfb9b656-jtls9"
Oct 31 04:39:28.764603 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:28.764551    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-79468f7c9f-559sl" podStartSLOduration=1.764538164 podStartE2EDuration="1.764538164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:28.763721432 +0000 UTC m=+5521.730054777" watchObservedRunningTime="2026-10-31 04:39:28.764538164 +0000 UTC m=+5521.730871498"
Oct 31 04:39:29.035079 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:39:29.035048    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd926de8d_6439_41a4_8ad6_1c01833ed69c.slice/crio-02a7fee526d9d5394b2930d9f2427493f6d9f8cff69dd61d04a431a1645ea02a WatchSource:0}: Error finding container 02a7fee526d9d5394b2930d9f2427493f6d9f8cff69dd61d04a431a1645ea02a: Status 404 returned error can't find the container with id 02a7fee526d9d5394b2930d9f2427493f6d9f8cff69dd61d04a431a1645ea02a
Oct 31 04:39:29.035543 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:29.035518    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-9cfb9b656-jtls9"]
Oct 31 04:39:29.654037 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:29.653997    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-9cfb9b656-jtls9" event={"ID":"d926de8d-6439-41a4-8ad6-1c01833ed69c","Type":"ContainerStarted","Data":"eb48cb598bb81f9ec09fe8e87f20a66a1a4d682ce99c82d9605bbd5bf3f8553b"}
Oct 31 04:39:29.654321 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:29.654060    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-9cfb9b656-jtls9" event={"ID":"d926de8d-6439-41a4-8ad6-1c01833ed69c","Type":"ContainerStarted","Data":"02a7fee526d9d5394b2930d9f2427493f6d9f8cff69dd61d04a431a1645ea02a"}
Oct 31 04:39:29.654321 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:29.654093    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-9cfb9b656-jtls9"
Oct 31 04:39:29.768326 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:29.768273    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-9cfb9b656-jtls9" podStartSLOduration=1.7682610140000001 podStartE2EDuration="1.768261014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:29.76704351 +0000 UTC m=+5522.733376854" watchObservedRunningTime="2026-10-31 04:39:29.768261014 +0000 UTC m=+5522.734594405"
Oct 31 04:39:30.750624 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:30.750582    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn" podUID="0b2b721d-414f-449f-ba75-918fb3a59972" containerName="payload-processing" containerID="cri-o://bc9be7a87afd89e4ba31fe71e5f4c837685247cd3a4e8abe301aa6638b44034b" gracePeriod=25
Oct 31 04:39:31.000420 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.000393    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"
Oct 31 04:39:31.148324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.148239    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b2b721d-414f-449f-ba75-918fb3a59972-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2b721d-414f-449f-ba75-918fb3a59972-plugins-config-volume\") pod \"0b2b721d-414f-449f-ba75-918fb3a59972\" (UID: \"0b2b721d-414f-449f-ba75-918fb3a59972\") "
Oct 31 04:39:31.148324 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.148307    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2b721d-414f-449f-ba75-918fb3a59972-kube-api-access-wqpqg\" (UniqueName: \"kubernetes.io/projected/0b2b721d-414f-449f-ba75-918fb3a59972-kube-api-access-wqpqg\") pod \"0b2b721d-414f-449f-ba75-918fb3a59972\" (UID: \"0b2b721d-414f-449f-ba75-918fb3a59972\") "
Oct 31 04:39:31.148644 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.148621    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b2b721d-414f-449f-ba75-918fb3a59972-plugins-config-volume" pod "0b2b721d-414f-449f-ba75-918fb3a59972" (UID: "0b2b721d-414f-449f-ba75-918fb3a59972"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:39:31.150627 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.150597    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2b721d-414f-449f-ba75-918fb3a59972-kube-api-access-wqpqg" pod "0b2b721d-414f-449f-ba75-918fb3a59972" (UID: "0b2b721d-414f-449f-ba75-918fb3a59972"). InnerVolumeSpecName "kube-api-access-wqpqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:39:31.249358 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.249325    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqpqg\" (UniqueName: \"kubernetes.io/projected/0b2b721d-414f-449f-ba75-918fb3a59972-kube-api-access-wqpqg\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:39:31.249530 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.249438    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2b721d-414f-449f-ba75-918fb3a59972-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:39:31.665986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.665946    4900 generic.go:356] "Generic (PLEG): container finished" podID="0b2b721d-414f-449f-ba75-918fb3a59972" containerID="bc9be7a87afd89e4ba31fe71e5f4c837685247cd3a4e8abe301aa6638b44034b" exitCode=0
Oct 31 04:39:31.666166 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.665996    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"
Oct 31 04:39:31.666166 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.666003    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn" event={"ID":"0b2b721d-414f-449f-ba75-918fb3a59972","Type":"ContainerDied","Data":"bc9be7a87afd89e4ba31fe71e5f4c837685247cd3a4e8abe301aa6638b44034b"}
Oct 31 04:39:31.666166 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.666049    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-k2qfn" event={"ID":"0b2b721d-414f-449f-ba75-918fb3a59972","Type":"ContainerDied","Data":"5f1e9c6f9c150f073b73bebb8aa2949489f508c58d104082ab198da94e226312"}
Oct 31 04:39:31.666166 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.666070    4900 scope.go:122] "RemoveContainer" containerID="bc9be7a87afd89e4ba31fe71e5f4c837685247cd3a4e8abe301aa6638b44034b"
Oct 31 04:39:31.674241 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.674222    4900 scope.go:122] "RemoveContainer" containerID="bc9be7a87afd89e4ba31fe71e5f4c837685247cd3a4e8abe301aa6638b44034b"
Oct 31 04:39:31.674521 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:39:31.674497    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc9be7a87afd89e4ba31fe71e5f4c837685247cd3a4e8abe301aa6638b44034b\": container with ID starting with bc9be7a87afd89e4ba31fe71e5f4c837685247cd3a4e8abe301aa6638b44034b not found: ID does not exist" containerID="bc9be7a87afd89e4ba31fe71e5f4c837685247cd3a4e8abe301aa6638b44034b"
Oct 31 04:39:31.674581 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.674530    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9be7a87afd89e4ba31fe71e5f4c837685247cd3a4e8abe301aa6638b44034b"} err="failed to get container status \"bc9be7a87afd89e4ba31fe71e5f4c837685247cd3a4e8abe301aa6638b44034b\": rpc error: code = NotFound desc = could not find container \"bc9be7a87afd89e4ba31fe71e5f4c837685247cd3a4e8abe301aa6638b44034b\": container with ID starting with bc9be7a87afd89e4ba31fe71e5f4c837685247cd3a4e8abe301aa6638b44034b not found: ID does not exist"
Oct 31 04:39:31.831196 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.831154    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"]
Oct 31 04:39:31.881245 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:31.881210    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-k2qfn"]
Oct 31 04:39:33.157361 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:33.157321    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b2b721d-414f-449f-ba75-918fb3a59972" path="/var/lib/kubelet/pods/0b2b721d-414f-449f-ba75-918fb3a59972/volumes"
Oct 31 04:39:36.983203 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:39:36.983168    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:39:39.657125 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:39.657092    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-79468f7c9f-559sl"
Oct 31 04:39:39.823819 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:39.823659    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-597dbd485d-sbjjw"]
Oct 31 04:39:40.662677 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:40.662646    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-9cfb9b656-jtls9"
Oct 31 04:39:40.827703 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:40.827670    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"]
Oct 31 04:39:43.287204 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:43.287161    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:43.739888 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:43.739862    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:44.824662 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:44.824612    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-597dbd485d-sbjjw" podUID="6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8" containerName="payload-processing" containerID="cri-o://dea6809d102fc0579a605f7e7d033fc4405f42bfeb26e2bbe6d04b862a74aad4" gracePeriod=25
Oct 31 04:39:45.071612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.071584    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-597dbd485d-sbjjw"
Oct 31 04:39:45.084215 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.084165    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8-plugins-config-volume\") pod \"6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8\" (UID: \"6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8\") "
Oct 31 04:39:45.084313 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.084289    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8-kube-api-access-jchl5\" (UniqueName: \"kubernetes.io/projected/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8-kube-api-access-jchl5\") pod \"6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8\" (UID: \"6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8\") "
Oct 31 04:39:45.084507 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.084488    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8-plugins-config-volume" pod "6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8" (UID: "6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:39:45.084622 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.084601    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:39:45.086362 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.086332    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8-kube-api-access-jchl5" pod "6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8" (UID: "6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8"). InnerVolumeSpecName "kube-api-access-jchl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:39:45.185910 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.185872    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jchl5\" (UniqueName: \"kubernetes.io/projected/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8-kube-api-access-jchl5\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:39:45.732638 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.732600    4900 generic.go:356] "Generic (PLEG): container finished" podID="6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8" containerID="dea6809d102fc0579a605f7e7d033fc4405f42bfeb26e2bbe6d04b862a74aad4" exitCode=0
Oct 31 04:39:45.732852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.732670    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-597dbd485d-sbjjw"
Oct 31 04:39:45.732852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.732680    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-597dbd485d-sbjjw" event={"ID":"6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8","Type":"ContainerDied","Data":"dea6809d102fc0579a605f7e7d033fc4405f42bfeb26e2bbe6d04b862a74aad4"}
Oct 31 04:39:45.732852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.732739    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-597dbd485d-sbjjw" event={"ID":"6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8","Type":"ContainerDied","Data":"158cfbb49dfcd64133d0049e2243a4539554cd29a312397778b55c09f3a9763b"}
Oct 31 04:39:45.732852 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.732763    4900 scope.go:122] "RemoveContainer" containerID="dea6809d102fc0579a605f7e7d033fc4405f42bfeb26e2bbe6d04b862a74aad4"
Oct 31 04:39:45.741056 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.741032    4900 scope.go:122] "RemoveContainer" containerID="dea6809d102fc0579a605f7e7d033fc4405f42bfeb26e2bbe6d04b862a74aad4"
Oct 31 04:39:45.741316 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:39:45.741300    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea6809d102fc0579a605f7e7d033fc4405f42bfeb26e2bbe6d04b862a74aad4\": container with ID starting with dea6809d102fc0579a605f7e7d033fc4405f42bfeb26e2bbe6d04b862a74aad4 not found: ID does not exist" containerID="dea6809d102fc0579a605f7e7d033fc4405f42bfeb26e2bbe6d04b862a74aad4"
Oct 31 04:39:45.741378 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.741325    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea6809d102fc0579a605f7e7d033fc4405f42bfeb26e2bbe6d04b862a74aad4"} err="failed to get container status \"dea6809d102fc0579a605f7e7d033fc4405f42bfeb26e2bbe6d04b862a74aad4\": rpc error: code = NotFound desc = could not find container \"dea6809d102fc0579a605f7e7d033fc4405f42bfeb26e2bbe6d04b862a74aad4\": container with ID starting with dea6809d102fc0579a605f7e7d033fc4405f42bfeb26e2bbe6d04b862a74aad4 not found: ID does not exist"
Oct 31 04:39:45.828089 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.828052    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw" podUID="7257aff6-06c1-40ff-9e8a-8a79e40307a0" containerName="payload-pre-processing" containerID="cri-o://f2b4f00a78bc440ab4152e0116859a3898b184ff60c7c065d8fb1af7aa9bf132" gracePeriod=25
Oct 31 04:39:45.896557 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.896520    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-597dbd485d-sbjjw"]
Oct 31 04:39:45.946388 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:45.946364    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-597dbd485d-sbjjw"]
Oct 31 04:39:46.076319 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.076294    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"
Oct 31 04:39:46.095023 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.094997    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7257aff6-06c1-40ff-9e8a-8a79e40307a0-kube-api-access-bzrbk\" (UniqueName: \"kubernetes.io/projected/7257aff6-06c1-40ff-9e8a-8a79e40307a0-kube-api-access-bzrbk\") pod \"7257aff6-06c1-40ff-9e8a-8a79e40307a0\" (UID: \"7257aff6-06c1-40ff-9e8a-8a79e40307a0\") "
Oct 31 04:39:46.095179 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.095116    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7257aff6-06c1-40ff-9e8a-8a79e40307a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7257aff6-06c1-40ff-9e8a-8a79e40307a0-plugins-config-volume\") pod \"7257aff6-06c1-40ff-9e8a-8a79e40307a0\" (UID: \"7257aff6-06c1-40ff-9e8a-8a79e40307a0\") "
Oct 31 04:39:46.095471 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.095451    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7257aff6-06c1-40ff-9e8a-8a79e40307a0-plugins-config-volume" pod "7257aff6-06c1-40ff-9e8a-8a79e40307a0" (UID: "7257aff6-06c1-40ff-9e8a-8a79e40307a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:39:46.097111 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.097091    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7257aff6-06c1-40ff-9e8a-8a79e40307a0-kube-api-access-bzrbk" pod "7257aff6-06c1-40ff-9e8a-8a79e40307a0" (UID: "7257aff6-06c1-40ff-9e8a-8a79e40307a0"). InnerVolumeSpecName "kube-api-access-bzrbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:39:46.196417 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.196386    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7257aff6-06c1-40ff-9e8a-8a79e40307a0-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:39:46.196417 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.196413    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzrbk\" (UniqueName: \"kubernetes.io/projected/7257aff6-06c1-40ff-9e8a-8a79e40307a0-kube-api-access-bzrbk\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:39:46.526535 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.526489    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:39:46.738949 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.738911    4900 generic.go:356] "Generic (PLEG): container finished" podID="7257aff6-06c1-40ff-9e8a-8a79e40307a0" containerID="f2b4f00a78bc440ab4152e0116859a3898b184ff60c7c065d8fb1af7aa9bf132" exitCode=0
Oct 31 04:39:46.739128 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.738956    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"
Oct 31 04:39:46.739128 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.738996    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw" event={"ID":"7257aff6-06c1-40ff-9e8a-8a79e40307a0","Type":"ContainerDied","Data":"f2b4f00a78bc440ab4152e0116859a3898b184ff60c7c065d8fb1af7aa9bf132"}
Oct 31 04:39:46.739128 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.739051    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw" event={"ID":"7257aff6-06c1-40ff-9e8a-8a79e40307a0","Type":"ContainerDied","Data":"99807a7567bca05e341965b404ec2dba5e815411be2031b68c572528c39846df"}
Oct 31 04:39:46.739128 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.739076    4900 scope.go:122] "RemoveContainer" containerID="f2b4f00a78bc440ab4152e0116859a3898b184ff60c7c065d8fb1af7aa9bf132"
Oct 31 04:39:46.747518 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.747499    4900 scope.go:122] "RemoveContainer" containerID="f2b4f00a78bc440ab4152e0116859a3898b184ff60c7c065d8fb1af7aa9bf132"
Oct 31 04:39:46.747773 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:39:46.747754    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2b4f00a78bc440ab4152e0116859a3898b184ff60c7c065d8fb1af7aa9bf132\": container with ID starting with f2b4f00a78bc440ab4152e0116859a3898b184ff60c7c065d8fb1af7aa9bf132 not found: ID does not exist" containerID="f2b4f00a78bc440ab4152e0116859a3898b184ff60c7c065d8fb1af7aa9bf132"
Oct 31 04:39:46.747876 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.747780    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2b4f00a78bc440ab4152e0116859a3898b184ff60c7c065d8fb1af7aa9bf132"} err="failed to get container status \"f2b4f00a78bc440ab4152e0116859a3898b184ff60c7c065d8fb1af7aa9bf132\": rpc error: code = NotFound desc = could not find container \"f2b4f00a78bc440ab4152e0116859a3898b184ff60c7c065d8fb1af7aa9bf132\": container with ID starting with f2b4f00a78bc440ab4152e0116859a3898b184ff60c7c065d8fb1af7aa9bf132 not found: ID does not exist"
Oct 31 04:39:46.901229 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.901108    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"]
Oct 31 04:39:46.951071 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:46.951038    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-6545dbfb5c-prvtw"]
Oct 31 04:39:47.152304 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.152230    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8" path="/var/lib/kubelet/pods/6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8/volumes"
Oct 31 04:39:47.152631 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.152618    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7257aff6-06c1-40ff-9e8a-8a79e40307a0" path="/var/lib/kubelet/pods/7257aff6-06c1-40ff-9e8a-8a79e40307a0/volumes"
Oct 31 04:39:47.596284 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.596250    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"]
Oct 31 04:39:47.596668 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.596653    4900 cpu_manager.go:414] "containerMap: removing container" podUID="6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8" containerName="payload-processing"
Oct 31 04:39:47.596732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.596671    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8" containerName="payload-processing"
Oct 31 04:39:47.596732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.596692    4900 cpu_manager.go:414] "containerMap: removing container" podUID="0b2b721d-414f-449f-ba75-918fb3a59972" containerName="payload-processing"
Oct 31 04:39:47.596732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.596697    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b2b721d-414f-449f-ba75-918fb3a59972" containerName="payload-processing"
Oct 31 04:39:47.596732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.596727    4900 cpu_manager.go:414] "containerMap: removing container" podUID="7257aff6-06c1-40ff-9e8a-8a79e40307a0" containerName="payload-pre-processing"
Oct 31 04:39:47.596901 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.596733    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7257aff6-06c1-40ff-9e8a-8a79e40307a0" containerName="payload-pre-processing"
Oct 31 04:39:47.596901 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.596812    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bfe5f98-7e9b-4c86-bfb7-c8d04815d9e8" containerName="payload-processing"
Oct 31 04:39:47.596901 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.596831    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b2b721d-414f-449f-ba75-918fb3a59972" containerName="payload-processing"
Oct 31 04:39:47.596901 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.596841    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="7257aff6-06c1-40ff-9e8a-8a79e40307a0" containerName="payload-pre-processing"
Oct 31 04:39:47.601068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.601052    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:47.653821 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.653774    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"]
Oct 31 04:39:47.701552 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.701509    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 04:39:47.701552 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.701510    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 04:39:47.701552 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.701510    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 04:39:47.701552 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.701516    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Oct 31 04:39:47.701848 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.701604    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-595zg\""
Oct 31 04:39:47.703498 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.703479    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 04:39:47.711987 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.711963    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-lqzd2\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:47.712115 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.711996    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pcpl\" (UniqueName: \"kubernetes.io/projected/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-kube-api-access-8pcpl\") pod \"payload-processing-external-model-5bcd7997-lqzd2\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:47.712115 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.712026    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-lqzd2\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:47.712191 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.712125    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-lqzd2\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:47.812845 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.812809    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-lqzd2\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:47.813039 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.812852    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pcpl\" (UniqueName: \"kubernetes.io/projected/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-kube-api-access-8pcpl\") pod \"payload-processing-external-model-5bcd7997-lqzd2\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:47.813039 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.812905    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-lqzd2\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:47.813039 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.812940    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-lqzd2\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:47.813632 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.813608    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-lqzd2\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:47.813740 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.813725    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-lqzd2\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:47.815282 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.815264    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-lqzd2\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:47.870233 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.870158    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pcpl\" (UniqueName: \"kubernetes.io/projected/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-kube-api-access-8pcpl\") pod \"payload-processing-external-model-5bcd7997-lqzd2\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:47.912945 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:47.912910    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:48.277184 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:39:48.277153    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dc14b7_93d5_4ae1_8ba2_d3bf7c44f49c.slice/crio-38303026e23265414398d0abcc45a5b2eb77cccabd00c060ed815f8be5399353 WatchSource:0}: Error finding container 38303026e23265414398d0abcc45a5b2eb77cccabd00c060ed815f8be5399353: Status 404 returned error can't find the container with id 38303026e23265414398d0abcc45a5b2eb77cccabd00c060ed815f8be5399353
Oct 31 04:39:48.278651 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:48.278624    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"]
Oct 31 04:39:48.754039 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:48.753981    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2" event={"ID":"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c","Type":"ContainerStarted","Data":"99397cac949fb44f47db82e7c12136942c50673ac1e0eeb13a76ce525771546e"}
Oct 31 04:39:48.754234 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:48.754060    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:39:48.754234 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:48.754086    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2" event={"ID":"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c","Type":"ContainerStarted","Data":"38303026e23265414398d0abcc45a5b2eb77cccabd00c060ed815f8be5399353"}
Oct 31 04:39:48.872716 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:48.872653    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2" podStartSLOduration=1.8726388699999998 podStartE2EDuration="1.87263887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:39:48.870937425 +0000 UTC m=+5541.837270770" watchObservedRunningTime="2026-10-31 04:39:48.87263887 +0000 UTC m=+5541.838972208"
Oct 31 04:39:59.775504 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:39:59.775468    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:40:06.982764 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:40:06.982724    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:40:13.998349 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:40:13.998319    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:15.832130 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:40:15.832094    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:40:36.984151 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:40:36.984116    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:41:06.983059 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:41:06.983016    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:41:14.311316 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:14.311278    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:14.525941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:14.525903    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:17.283598 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:17.283555    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:17.496274 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:17.496239    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:29.443537 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:29.443501    4900 scope.go:122] "RemoveContainer" containerID="f13e2c48ecd14799e05097f6b0e1ece8588b6366a992d0f7973a680804f0513a"
Oct 31 04:41:36.983616 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:41:36.983572    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:41:43.040143 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:43.040108    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"]
Oct 31 04:41:43.286922 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:43.286889    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:43.515978 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:43.515945    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:43.806072 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:43.805989    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:41:48.044813 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.041860    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2" podUID="f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c" containerName="payload-processing" containerID="cri-o://99397cac949fb44f47db82e7c12136942c50673ac1e0eeb13a76ce525771546e" gracePeriod=25
Oct 31 04:41:48.335400 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.335154    4900 generic.go:356] "Generic (PLEG): container finished" podID="f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c" containerID="99397cac949fb44f47db82e7c12136942c50673ac1e0eeb13a76ce525771546e" exitCode=0
Oct 31 04:41:48.335400 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.335327    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2" event={"ID":"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c","Type":"ContainerDied","Data":"99397cac949fb44f47db82e7c12136942c50673ac1e0eeb13a76ce525771546e"}
Oct 31 04:41:48.411842 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.411812    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:41:48.524132 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.524096    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-kube-api-access-8pcpl\" (UniqueName: \"kubernetes.io/projected/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-kube-api-access-8pcpl\") pod \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") "
Oct 31 04:41:48.524446 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.524410    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-plugins-config-volume\") pod \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") "
Oct 31 04:41:48.524563 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.524524    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-routing-overlay-volume\") pod \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") "
Oct 31 04:41:48.524618 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.524598    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-provider-credentials\") pod \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\" (UID: \"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c\") "
Oct 31 04:41:48.524912 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.524875    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-plugins-config-volume" pod "f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c" (UID: "f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:41:48.525172 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.525069    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:41:48.525172 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.525160    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-routing-overlay-volume" pod "f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c" (UID: "f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:41:48.537407 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.528719    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-kube-api-access-8pcpl" pod "f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c" (UID: "f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c"). InnerVolumeSpecName "kube-api-access-8pcpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:41:48.542938 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.542900    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-provider-credentials" pod "f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c" (UID: "f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:41:48.626103 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.626023    4900 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-provider-credentials\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:41:48.626103 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.626051    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pcpl\" (UniqueName: \"kubernetes.io/projected/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-kube-api-access-8pcpl\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:41:48.626103 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:48.626062    4900 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c-routing-overlay-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:41:49.341666 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:49.341631    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2" event={"ID":"f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c","Type":"ContainerDied","Data":"38303026e23265414398d0abcc45a5b2eb77cccabd00c060ed815f8be5399353"}
Oct 31 04:41:49.342139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:49.341676    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"
Oct 31 04:41:49.342139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:49.341701    4900 scope.go:122] "RemoveContainer" containerID="99397cac949fb44f47db82e7c12136942c50673ac1e0eeb13a76ce525771546e"
Oct 31 04:41:49.504684 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:49.504652    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"]
Oct 31 04:41:49.554807 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:49.554751    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-lqzd2"]
Oct 31 04:41:51.151988 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:41:51.151953    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c" path="/var/lib/kubelet/pods/f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c/volumes"
Oct 31 04:42:06.983927 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:42:06.983886    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:42:14.660910 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:42:14.660831    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:42:29.075994 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:42:29.075960    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:42:29.076542 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:42:29.076379    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:42:36.983509 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:42:36.983467    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:43:06.984308 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:43:06.984266    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:43:15.030526 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:43:15.030485    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:29.147028 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:43:29.146992    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:43:36.983673 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:43:36.983626    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:44:06.983074 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:44:06.983036    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:44:15.302461 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:44:15.302422    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:15.523568 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:44:15.523540    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:44:36.983293 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:44:36.983255    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:45:06.983332 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:45:06.983234    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:45:15.630390 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:45:15.630284    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:28.873936 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:45:28.873884    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:36.982966 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:45:36.982928    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:45:43.105581 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:45:43.105541    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:45:43.438619 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:45:43.438521    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:06.983278 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:46:06.983242    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:46:15.959444 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:46:15.959408    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:46:36.983377 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:46:36.983339    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:47:04.376340 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:04.376302    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:05.000062 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:05.000033    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:05.212431 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:05.212387    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:06.983805 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:47:06.983754    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:47:09.878400 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:09.878349    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:10.098526 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:10.098490    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:12.896979 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:12.896940    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:13.117136 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.117102    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:13.241311 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.241277    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"]
Oct 31 04:47:13.241816 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.241767    4900 cpu_manager.go:414] "containerMap: removing container" podUID="f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c" containerName="payload-processing"
Oct 31 04:47:13.241816 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.241801    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c" containerName="payload-processing"
Oct 31 04:47:13.241996 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.241908    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4dc14b7-93d5-4ae1-8ba2-d3bf7c44f49c" containerName="payload-processing"
Oct 31 04:47:13.245089 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.245071    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"
Oct 31 04:47:13.297496 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.297456    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"]
Oct 31 04:47:13.305936 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.305893    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb\" (UID: \"f06b87bd-ba8a-42cc-a4ef-0b6156b425a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"
Oct 31 04:47:13.306124 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.306010    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pjcf\" (UniqueName: \"kubernetes.io/projected/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0-kube-api-access-4pjcf\") pod \"payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb\" (UID: \"f06b87bd-ba8a-42cc-a4ef-0b6156b425a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"
Oct 31 04:47:13.345373 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.345342    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c1662147-dockercfg-d8pw9\""
Oct 31 04:47:13.352484 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.352459    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-cc1991\""
Oct 31 04:47:13.394603 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.394572    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"]
Oct 31 04:47:13.398035 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.398019    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"
Oct 31 04:47:13.406894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.406859    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb\" (UID: \"f06b87bd-ba8a-42cc-a4ef-0b6156b425a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"
Oct 31 04:47:13.407022 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.406919    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pjcf\" (UniqueName: \"kubernetes.io/projected/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0-kube-api-access-4pjcf\") pod \"payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb\" (UID: \"f06b87bd-ba8a-42cc-a4ef-0b6156b425a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"
Oct 31 04:47:13.407568 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.407547    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb\" (UID: \"f06b87bd-ba8a-42cc-a4ef-0b6156b425a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"
Oct 31 04:47:13.450034 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.450001    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"]
Oct 31 04:47:13.463479 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.463445    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pjcf\" (UniqueName: \"kubernetes.io/projected/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0-kube-api-access-4pjcf\") pod \"payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb\" (UID: \"f06b87bd-ba8a-42cc-a4ef-0b6156b425a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"
Oct 31 04:47:13.507887 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.507772    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76edd686-a4f9-4da9-914d-01362ea325a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j\" (UID: \"76edd686-a4f9-4da9-914d-01362ea325a0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"
Oct 31 04:47:13.508050 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.507985    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzmqg\" (UniqueName: \"kubernetes.io/projected/76edd686-a4f9-4da9-914d-01362ea325a0-kube-api-access-qzmqg\") pod \"payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j\" (UID: \"76edd686-a4f9-4da9-914d-01362ea325a0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"
Oct 31 04:47:13.557924 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.557888    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"
Oct 31 04:47:13.608655 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.608623    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzmqg\" (UniqueName: \"kubernetes.io/projected/76edd686-a4f9-4da9-914d-01362ea325a0-kube-api-access-qzmqg\") pod \"payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j\" (UID: \"76edd686-a4f9-4da9-914d-01362ea325a0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"
Oct 31 04:47:13.608845 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.608719    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76edd686-a4f9-4da9-914d-01362ea325a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j\" (UID: \"76edd686-a4f9-4da9-914d-01362ea325a0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"
Oct 31 04:47:13.609641 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.609617    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76edd686-a4f9-4da9-914d-01362ea325a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j\" (UID: \"76edd686-a4f9-4da9-914d-01362ea325a0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"
Oct 31 04:47:13.665671 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.665633    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzmqg\" (UniqueName: \"kubernetes.io/projected/76edd686-a4f9-4da9-914d-01362ea325a0-kube-api-access-qzmqg\") pod \"payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j\" (UID: \"76edd686-a4f9-4da9-914d-01362ea325a0\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"
Oct 31 04:47:13.711627 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.711591    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"
Oct 31 04:47:13.921004 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:47:13.920963    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06b87bd_ba8a_42cc_a4ef_0b6156b425a0.slice/crio-f3a7d03ac26127deff707e87d788151308653ec30836f4eee928be92b22c245f WatchSource:0}: Error finding container f3a7d03ac26127deff707e87d788151308653ec30836f4eee928be92b22c245f: Status 404 returned error can't find the container with id f3a7d03ac26127deff707e87d788151308653ec30836f4eee928be92b22c245f
Oct 31 04:47:13.923078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:13.923052    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"]
Oct 31 04:47:14.074781 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:47:14.074750    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76edd686_a4f9_4da9_914d_01362ea325a0.slice/crio-c4e0aca464cc684056a7de77a6e36deb463b0cb73db8807b817e070fc58662df WatchSource:0}: Error finding container c4e0aca464cc684056a7de77a6e36deb463b0cb73db8807b817e070fc58662df: Status 404 returned error can't find the container with id c4e0aca464cc684056a7de77a6e36deb463b0cb73db8807b817e070fc58662df
Oct 31 04:47:14.077065 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:14.077044    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"]
Oct 31 04:47:14.904589 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:14.904556    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb" event={"ID":"f06b87bd-ba8a-42cc-a4ef-0b6156b425a0","Type":"ContainerStarted","Data":"ba8e922a34da2ff224ad1215a847db9913204d5c90e6fc108789fa83f867d73c"}
Oct 31 04:47:14.904828 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:14.904606    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb" event={"ID":"f06b87bd-ba8a-42cc-a4ef-0b6156b425a0","Type":"ContainerStarted","Data":"f3a7d03ac26127deff707e87d788151308653ec30836f4eee928be92b22c245f"}
Oct 31 04:47:14.904828 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:14.904665    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"
Oct 31 04:47:14.906194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:14.906166    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j" event={"ID":"76edd686-a4f9-4da9-914d-01362ea325a0","Type":"ContainerStarted","Data":"f5b243b0dc65c275163b2130f833a379f70ae955e20c0599cc492561652af296"}
Oct 31 04:47:14.906296 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:14.906214    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j" event={"ID":"76edd686-a4f9-4da9-914d-01362ea325a0","Type":"ContainerStarted","Data":"c4e0aca464cc684056a7de77a6e36deb463b0cb73db8807b817e070fc58662df"}
Oct 31 04:47:14.906296 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:14.906239    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"
Oct 31 04:47:15.018332 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.018271    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb" podStartSLOduration=2.01825617 podStartE2EDuration="2.01825617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:15.016967569 +0000 UTC m=+5987.983300913" watchObservedRunningTime="2026-10-31 04:47:15.01825617 +0000 UTC m=+5987.984589514"
Oct 31 04:47:15.130856 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.130800    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j" podStartSLOduration=2.130767943 podStartE2EDuration="2.130767943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:15.128338358 +0000 UTC m=+5988.094671717" watchObservedRunningTime="2026-10-31 04:47:15.130767943 +0000 UTC m=+5988.097101305"
Oct 31 04:47:15.522602 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.522559    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"]
Oct 31 04:47:15.526265 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.526238    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:15.579389 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.579356    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"]
Oct 31 04:47:15.627139 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.627104    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-cc1991-serving-cert\""
Oct 31 04:47:15.627537 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.627521    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vgcd\" (UniqueName: \"kubernetes.io/projected/34cc281c-75e7-43e4-a935-afbaa9be505d-kube-api-access-2vgcd\") pod \"maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:15.627582 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.627570    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:15.627677 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.627656    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34cc281c-75e7-43e4-a935-afbaa9be505d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:15.627736 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.627700    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:15.728349 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.728293    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:15.728568 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.728360    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34cc281c-75e7-43e4-a935-afbaa9be505d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:15.728568 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.728390    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:15.728568 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.728474    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vgcd\" (UniqueName: \"kubernetes.io/projected/34cc281c-75e7-43e4-a935-afbaa9be505d-kube-api-access-2vgcd\") pod \"maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:15.728747 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:47:15.728557    4900 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-serving-cert: secret "maas-api-e2e-worker-single-main-cc1991-serving-cert" not found
Oct 31 04:47:15.728747 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:47:15.728651    4900 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-maas-api-tls podName:34cc281c-75e7-43e4-a935-afbaa9be505d nodeName:}" failed. No retries permitted until 2026-10-31 04:47:16.228629963 +0000 UTC m=+5989.194963301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-maas-api-tls") pod "maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn" (UID: "34cc281c-75e7-43e4-a935-afbaa9be505d") : secret "maas-api-e2e-worker-single-main-cc1991-serving-cert" not found
Oct 31 04:47:15.729174 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.729152    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34cc281c-75e7-43e4-a935-afbaa9be505d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:15.730777 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.730759    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:15.785777 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.785703    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vgcd\" (UniqueName: \"kubernetes.io/projected/34cc281c-75e7-43e4-a935-afbaa9be505d-kube-api-access-2vgcd\") pod \"maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:15.790070 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.790042    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:15.997554 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:15.997516    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:16.214193 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:16.214111    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:16.234587 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:16.234552    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:16.237051 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:16.237028    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:16.415990 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:16.415953    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:16.439335 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:16.439305    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:16.803916 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:47:16.803878    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34cc281c_75e7_43e4_a935_afbaa9be505d.slice/crio-5a6d999aeccdd170fb132e5522bd2ce08b7eea664d4124e68b7e8b70975cab41 WatchSource:0}: Error finding container 5a6d999aeccdd170fb132e5522bd2ce08b7eea664d4124e68b7e8b70975cab41: Status 404 returned error can't find the container with id 5a6d999aeccdd170fb132e5522bd2ce08b7eea664d4124e68b7e8b70975cab41
Oct 31 04:47:16.805604 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:16.805579    4900 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:47:16.805713 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:16.805692    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"]
Oct 31 04:47:16.917041 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:16.917004    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn" event={"ID":"34cc281c-75e7-43e4-a935-afbaa9be505d","Type":"ContainerStarted","Data":"5a6d999aeccdd170fb132e5522bd2ce08b7eea664d4124e68b7e8b70975cab41"}
Oct 31 04:47:17.923764 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:17.923728    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn" event={"ID":"34cc281c-75e7-43e4-a935-afbaa9be505d","Type":"ContainerStarted","Data":"5a9efa7a21d0610e88d619ecd04cac15a984bdbcc6955b43bd8e544091e5d453"}
Oct 31 04:47:17.924255 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:17.923926    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:18.063574 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:18.063517    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn" podStartSLOduration=2.027423355 podStartE2EDuration="3.063502822s" totalImagesPullingTime="1.036079467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:18.061633591 +0000 UTC m=+5991.027966935" watchObservedRunningTime="2026-10-31 04:47:18.063502822 +0000 UTC m=+5991.029836165"
Oct 31 04:47:20.333204 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:20.333159    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:23.934187 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:23.934160    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:47:24.866835 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:24.866799    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:25.914475 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:25.914446    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"
Oct 31 04:47:25.914922 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:25.914521    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"
Oct 31 04:47:29.108538 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:29.108506    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:47:29.110989 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:29.110965    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:47:29.331649 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:29.331617    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:34.062389 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:34.062358    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:34.762727 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:34.762693    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:35.349192 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:35.349154    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:35.619107 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:35.619024    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:36.983909 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:47:36.983874    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:47:38.392333 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.392296    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"]
Oct 31 04:47:38.397174 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.397152    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.452067 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.452039    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-cc1991\"/\"openshift-service-ca.crt\""
Oct 31 04:47:38.452244 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.452077    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-cc1991\"/\"kube-root-ca.crt\""
Oct 31 04:47:38.502181 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.502145    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-cc1991\"/\"default-dockercfg-c5kxz\""
Oct 31 04:47:38.502374 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.502277    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-cc1991\"/\"simulator-single-main-cc1991-kserve-self-signed-certs\""
Oct 31 04:47:38.504000 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.503978    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"]
Oct 31 04:47:38.562814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.562763    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-home\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.563004 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.562913    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b88d320-6ade-4021-8944-768c3e68c1b7-tls-certs\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.563054 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.563012    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-model-cache\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.563092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.563052    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-dshm\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.563128 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.563103    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxt7q\" (UniqueName: \"kubernetes.io/projected/4b88d320-6ade-4021-8944-768c3e68c1b7-kube-api-access-fxt7q\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.563169 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.563124    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-tmp-dir\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.606016 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.605985    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:38.663900 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.663810    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxt7q\" (UniqueName: \"kubernetes.io/projected/4b88d320-6ade-4021-8944-768c3e68c1b7-kube-api-access-fxt7q\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.663900 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.663859    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-tmp-dir\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.664097 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.663906    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-home\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.664097 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.664056    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b88d320-6ade-4021-8944-768c3e68c1b7-tls-certs\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.664177 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.664165    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-model-cache\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.664225 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.664210    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-dshm\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.664348 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.664330    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-home\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.664389 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.664347    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-tmp-dir\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.664542 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.664519    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-model-cache\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.666322 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.666306    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-dshm\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.666591 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.666575    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b88d320-6ade-4021-8944-768c3e68c1b7-tls-certs\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.720623 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.720583    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxt7q\" (UniqueName: \"kubernetes.io/projected/4b88d320-6ade-4021-8944-768c3e68c1b7-kube-api-access-fxt7q\") pod \"simulator-single-main-cc1991-kserve-64b869c776-8lpvz\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") " pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:38.834172 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:38.834134    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:39.008776 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:39.008742    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:39.348463 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:39.348376    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:39.380371 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:47:39.380344    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b88d320_6ade_4021_8944_768c3e68c1b7.slice/crio-685e9ad3565218704ab2bb2a0a1ee08a0164009f9d6dc6e45b457239c32b956f WatchSource:0}: Error finding container 685e9ad3565218704ab2bb2a0a1ee08a0164009f9d6dc6e45b457239c32b956f: Status 404 returned error can't find the container with id 685e9ad3565218704ab2bb2a0a1ee08a0164009f9d6dc6e45b457239c32b956f
Oct 31 04:47:39.382033 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:39.382013    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"]
Oct 31 04:47:40.022783 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:40.022746    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz" event={"ID":"4b88d320-6ade-4021-8944-768c3e68c1b7","Type":"ContainerStarted","Data":"9758828edeb0c310fe96c6e6762c3daeb90ed3ed8bea88b3a58a23647d3f1bd5"}
Oct 31 04:47:40.022783 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:40.022799    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz" event={"ID":"4b88d320-6ade-4021-8944-768c3e68c1b7","Type":"ContainerStarted","Data":"685e9ad3565218704ab2bb2a0a1ee08a0164009f9d6dc6e45b457239c32b956f"}
Oct 31 04:47:40.140052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:40.140000    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz" podStartSLOduration=2.139987776 podStartE2EDuration="2.139987776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:47:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:47:40.138379441 +0000 UTC m=+6013.104712786" watchObservedRunningTime="2026-10-31 04:47:40.139987776 +0000 UTC m=+6013.106321120"
Oct 31 04:47:41.009739 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:41.009701    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:43.146941 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:43.146907    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:47.626446 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:47.626405    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:49.009612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:49.009569    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:49.022493 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:49.022464    4900 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:49.077212 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:49.077180    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:47:52.148398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:52.148357    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:47:56.535970 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:47:56.535933    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:01.026751 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:01.026715    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:05.440035 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:05.440003    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:06.983830 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:48:06.983777    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:48:09.951769 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:09.951684    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:14.346540 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:14.346509    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:16.643694 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:16.643656    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:18.766052 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:18.766018    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:23.237416 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:23.237379    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:27.723592 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:27.723559    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:32.146253 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:32.146198    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:34.282969 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:34.282936    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:34.720506 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:34.720414    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:36.682037 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:36.681998    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:36.983603 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:48:36.983564    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:48:39.168660 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:39.168626    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"]
Oct 31 04:48:39.314427 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:39.314395    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:40.204843 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:40.204808    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"]
Oct 31 04:48:40.309728 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:40.309696    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"]
Oct 31 04:48:40.399439 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:40.399407    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:40.608204 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:40.608170    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:40.808467 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:40.808435    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:41.324305 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:41.324265    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:45.205724 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:45.205672    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j" podUID="76edd686-a4f9-4da9-914d-01362ea325a0" containerName="payload-processing" containerID="cri-o://f5b243b0dc65c275163b2130f833a379f70ae955e20c0599cc492561652af296" gracePeriod=25
Oct 31 04:48:45.310569 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:45.310521    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb" podUID="f06b87bd-ba8a-42cc-a4ef-0b6156b425a0" containerName="payload-pre-processing" containerID="cri-o://ba8e922a34da2ff224ad1215a847db9913204d5c90e6fc108789fa83f867d73c" gracePeriod=25
Oct 31 04:48:45.442889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:45.442851    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"]
Oct 31 04:48:45.443119 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:45.443093    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn" podUID="34cc281c-75e7-43e4-a935-afbaa9be505d" containerName="maas-api" containerID="cri-o://5a9efa7a21d0610e88d619ecd04cac15a984bdbcc6955b43bd8e544091e5d453" gracePeriod=30
Oct 31 04:48:45.611159 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:45.611127    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:45.913942 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:45.913854    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j" podUID="76edd686-a4f9-4da9-914d-01362ea325a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:48:45.913942 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:45.913879    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb" podUID="f06b87bd-ba8a-42cc-a4ef-0b6156b425a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:48:46.123664 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:46.123634    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:46.689986 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:46.689955    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:48:46.782869 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:46.782754    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34cc281c-75e7-43e4-a935-afbaa9be505d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34cc281c-75e7-43e4-a935-afbaa9be505d-openshift-service-ca\") pod \"34cc281c-75e7-43e4-a935-afbaa9be505d\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") "
Oct 31 04:48:46.782869 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:46.782862    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-maas-api-tls\") pod \"34cc281c-75e7-43e4-a935-afbaa9be505d\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") "
Oct 31 04:48:46.783066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:46.782995    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34cc281c-75e7-43e4-a935-afbaa9be505d-kube-api-access-2vgcd\" (UniqueName: \"kubernetes.io/projected/34cc281c-75e7-43e4-a935-afbaa9be505d-kube-api-access-2vgcd\") pod \"34cc281c-75e7-43e4-a935-afbaa9be505d\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") "
Oct 31 04:48:46.783066 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:46.783025    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-metrics-certs\") pod \"34cc281c-75e7-43e4-a935-afbaa9be505d\" (UID: \"34cc281c-75e7-43e4-a935-afbaa9be505d\") "
Oct 31 04:48:46.783213 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:46.783192    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34cc281c-75e7-43e4-a935-afbaa9be505d-openshift-service-ca" pod "34cc281c-75e7-43e4-a935-afbaa9be505d" (UID: "34cc281c-75e7-43e4-a935-afbaa9be505d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:48:46.783416 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:46.783401    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34cc281c-75e7-43e4-a935-afbaa9be505d-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:48:46.785183 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:46.785157    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34cc281c-75e7-43e4-a935-afbaa9be505d-kube-api-access-2vgcd" pod "34cc281c-75e7-43e4-a935-afbaa9be505d" (UID: "34cc281c-75e7-43e4-a935-afbaa9be505d"). InnerVolumeSpecName "kube-api-access-2vgcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:46.785296 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:46.785186    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-maas-api-tls" pod "34cc281c-75e7-43e4-a935-afbaa9be505d" (UID: "34cc281c-75e7-43e4-a935-afbaa9be505d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:48:46.785296 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:46.785201    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-metrics-certs" pod "34cc281c-75e7-43e4-a935-afbaa9be505d" (UID: "34cc281c-75e7-43e4-a935-afbaa9be505d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:48:46.884277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:46.884240    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:48:46.884277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:46.884275    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vgcd\" (UniqueName: \"kubernetes.io/projected/34cc281c-75e7-43e4-a935-afbaa9be505d-kube-api-access-2vgcd\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:48:46.884499 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:46.884286    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34cc281c-75e7-43e4-a935-afbaa9be505d-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:48:47.341305 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:47.341259    4900 generic.go:356] "Generic (PLEG): container finished" podID="34cc281c-75e7-43e4-a935-afbaa9be505d" containerID="5a9efa7a21d0610e88d619ecd04cac15a984bdbcc6955b43bd8e544091e5d453" exitCode=0
Oct 31 04:48:47.341493 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:47.341335    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"
Oct 31 04:48:47.341493 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:47.341365    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn" event={"ID":"34cc281c-75e7-43e4-a935-afbaa9be505d","Type":"ContainerDied","Data":"5a9efa7a21d0610e88d619ecd04cac15a984bdbcc6955b43bd8e544091e5d453"}
Oct 31 04:48:47.341493 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:47.341439    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn" event={"ID":"34cc281c-75e7-43e4-a935-afbaa9be505d","Type":"ContainerDied","Data":"5a6d999aeccdd170fb132e5522bd2ce08b7eea664d4124e68b7e8b70975cab41"}
Oct 31 04:48:47.341493 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:47.341465    4900 scope.go:122] "RemoveContainer" containerID="5a9efa7a21d0610e88d619ecd04cac15a984bdbcc6955b43bd8e544091e5d453"
Oct 31 04:48:47.350251 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:47.350232    4900 scope.go:122] "RemoveContainer" containerID="5a9efa7a21d0610e88d619ecd04cac15a984bdbcc6955b43bd8e544091e5d453"
Oct 31 04:48:47.350535 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:48:47.350513    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9efa7a21d0610e88d619ecd04cac15a984bdbcc6955b43bd8e544091e5d453\": container with ID starting with 5a9efa7a21d0610e88d619ecd04cac15a984bdbcc6955b43bd8e544091e5d453 not found: ID does not exist" containerID="5a9efa7a21d0610e88d619ecd04cac15a984bdbcc6955b43bd8e544091e5d453"
Oct 31 04:48:47.350581 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:47.350544    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9efa7a21d0610e88d619ecd04cac15a984bdbcc6955b43bd8e544091e5d453"} err="failed to get container status \"5a9efa7a21d0610e88d619ecd04cac15a984bdbcc6955b43bd8e544091e5d453\": rpc error: code = NotFound desc = could not find container \"5a9efa7a21d0610e88d619ecd04cac15a984bdbcc6955b43bd8e544091e5d453\": container with ID starting with 5a9efa7a21d0610e88d619ecd04cac15a984bdbcc6955b43bd8e544091e5d453 not found: ID does not exist"
Oct 31 04:48:47.504205 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:47.504167    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"]
Oct 31 04:48:47.554187 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:47.554151    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-cc1991-7c8769d84b-7f7rn"]
Oct 31 04:48:49.151682 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:49.151647    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34cc281c-75e7-43e4-a935-afbaa9be505d" path="/var/lib/kubelet/pods/34cc281c-75e7-43e4-a935-afbaa9be505d/volumes"
Oct 31 04:48:51.997410 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:51.997337    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:52.207501 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:52.207467    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:48:54.188714 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.188664    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz" podUID="4b88d320-6ade-4021-8944-768c3e68c1b7" containerName="main" containerID="cri-o://9758828edeb0c310fe96c6e6762c3daeb90ed3ed8bea88b3a58a23647d3f1bd5" gracePeriod=45
Oct 31 04:48:54.375047 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.375008    4900 generic.go:356] "Generic (PLEG): container finished" podID="4b88d320-6ade-4021-8944-768c3e68c1b7" containerID="9758828edeb0c310fe96c6e6762c3daeb90ed3ed8bea88b3a58a23647d3f1bd5" exitCode=0
Oct 31 04:48:54.375047 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.375035    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz" event={"ID":"4b88d320-6ade-4021-8944-768c3e68c1b7","Type":"ContainerDied","Data":"9758828edeb0c310fe96c6e6762c3daeb90ed3ed8bea88b3a58a23647d3f1bd5"}
Oct 31 04:48:54.440046 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.439978    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:48:54.563663 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.563631    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-model-cache\") pod \"4b88d320-6ade-4021-8944-768c3e68c1b7\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") "
Oct 31 04:48:54.563894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.563673    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-home\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-home\") pod \"4b88d320-6ade-4021-8944-768c3e68c1b7\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") "
Oct 31 04:48:54.563894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.563694    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-dshm\") pod \"4b88d320-6ade-4021-8944-768c3e68c1b7\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") "
Oct 31 04:48:54.563894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.563715    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-tmp-dir\") pod \"4b88d320-6ade-4021-8944-768c3e68c1b7\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") "
Oct 31 04:48:54.563894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.563805    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b88d320-6ade-4021-8944-768c3e68c1b7-kube-api-access-fxt7q\" (UniqueName: \"kubernetes.io/projected/4b88d320-6ade-4021-8944-768c3e68c1b7-kube-api-access-fxt7q\") pod \"4b88d320-6ade-4021-8944-768c3e68c1b7\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") "
Oct 31 04:48:54.563894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.563841    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b88d320-6ade-4021-8944-768c3e68c1b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b88d320-6ade-4021-8944-768c3e68c1b7-tls-certs\") pod \"4b88d320-6ade-4021-8944-768c3e68c1b7\" (UID: \"4b88d320-6ade-4021-8944-768c3e68c1b7\") "
Oct 31 04:48:54.564647 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.563979    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-home" pod "4b88d320-6ade-4021-8944-768c3e68c1b7" (UID: "4b88d320-6ade-4021-8944-768c3e68c1b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:48:54.564647 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.564064    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-tmp-dir" pod "4b88d320-6ade-4021-8944-768c3e68c1b7" (UID: "4b88d320-6ade-4021-8944-768c3e68c1b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:48:54.564647 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.564344    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-model-cache" pod "4b88d320-6ade-4021-8944-768c3e68c1b7" (UID: "4b88d320-6ade-4021-8944-768c3e68c1b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:48:54.570730 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.570696    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b88d320-6ade-4021-8944-768c3e68c1b7-tls-certs" pod "4b88d320-6ade-4021-8944-768c3e68c1b7" (UID: "4b88d320-6ade-4021-8944-768c3e68c1b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:48:54.570730 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.570719    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b88d320-6ade-4021-8944-768c3e68c1b7-kube-api-access-fxt7q" pod "4b88d320-6ade-4021-8944-768c3e68c1b7" (UID: "4b88d320-6ade-4021-8944-768c3e68c1b7"). InnerVolumeSpecName "kube-api-access-fxt7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:48:54.570730 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.570718    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-dshm" pod "4b88d320-6ade-4021-8944-768c3e68c1b7" (UID: "4b88d320-6ade-4021-8944-768c3e68c1b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:48:54.665746 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.665707    4900 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-model-cache\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:48:54.665746 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.665739    4900 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-home\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:48:54.665746 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.665749    4900 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-dshm\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:48:54.666018 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.665779    4900 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b88d320-6ade-4021-8944-768c3e68c1b7-tmp-dir\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:48:54.666018 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.665805    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxt7q\" (UniqueName: \"kubernetes.io/projected/4b88d320-6ade-4021-8944-768c3e68c1b7-kube-api-access-fxt7q\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:48:54.666018 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:54.665821    4900 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b88d320-6ade-4021-8944-768c3e68c1b7-tls-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:48:55.381572 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:55.381540    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"
Oct 31 04:48:55.381572 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:55.381555    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz" event={"ID":"4b88d320-6ade-4021-8944-768c3e68c1b7","Type":"ContainerDied","Data":"685e9ad3565218704ab2bb2a0a1ee08a0164009f9d6dc6e45b457239c32b956f"}
Oct 31 04:48:55.382087 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:55.381604    4900 scope.go:122] "RemoveContainer" containerID="9758828edeb0c310fe96c6e6762c3daeb90ed3ed8bea88b3a58a23647d3f1bd5"
Oct 31 04:48:55.545318 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:55.545284    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"]
Oct 31 04:48:55.595157 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:55.595121    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-cc1991/simulator-single-main-cc1991-kserve-64b869c776-8lpvz"]
Oct 31 04:48:55.913018 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:55.912925    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb" podUID="f06b87bd-ba8a-42cc-a4ef-0b6156b425a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:48:55.913175 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:55.912940    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j" podUID="76edd686-a4f9-4da9-914d-01362ea325a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:48:57.151821 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:48:57.151771    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b88d320-6ade-4021-8944-768c3e68c1b7" path="/var/lib/kubelet/pods/4b88d320-6ade-4021-8944-768c3e68c1b7/volumes"
Oct 31 04:49:05.430481 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.430450    4900 generic.go:356] "Generic (PLEG): container finished" podID="f06b87bd-ba8a-42cc-a4ef-0b6156b425a0" containerID="ba8e922a34da2ff224ad1215a847db9913204d5c90e6fc108789fa83f867d73c" exitCode=0
Oct 31 04:49:05.430892 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.430494    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb" event={"ID":"f06b87bd-ba8a-42cc-a4ef-0b6156b425a0","Type":"ContainerDied","Data":"ba8e922a34da2ff224ad1215a847db9913204d5c90e6fc108789fa83f867d73c"}
Oct 31 04:49:05.432249 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.432226    4900 generic.go:356] "Generic (PLEG): container finished" podID="76edd686-a4f9-4da9-914d-01362ea325a0" containerID="f5b243b0dc65c275163b2130f833a379f70ae955e20c0599cc492561652af296" exitCode=0
Oct 31 04:49:05.432361 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.432298    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j" event={"ID":"76edd686-a4f9-4da9-914d-01362ea325a0","Type":"ContainerDied","Data":"f5b243b0dc65c275163b2130f833a379f70ae955e20c0599cc492561652af296"}
Oct 31 04:49:05.464156 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.464091    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"
Oct 31 04:49:05.559814 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.559771    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"
Oct 31 04:49:05.581191 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.580299    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76edd686-a4f9-4da9-914d-01362ea325a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76edd686-a4f9-4da9-914d-01362ea325a0-plugins-config-volume\") pod \"76edd686-a4f9-4da9-914d-01362ea325a0\" (UID: \"76edd686-a4f9-4da9-914d-01362ea325a0\") "
Oct 31 04:49:05.581191 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.580399    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76edd686-a4f9-4da9-914d-01362ea325a0-kube-api-access-qzmqg\" (UniqueName: \"kubernetes.io/projected/76edd686-a4f9-4da9-914d-01362ea325a0-kube-api-access-qzmqg\") pod \"76edd686-a4f9-4da9-914d-01362ea325a0\" (UID: \"76edd686-a4f9-4da9-914d-01362ea325a0\") "
Oct 31 04:49:05.588091 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.588050    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76edd686-a4f9-4da9-914d-01362ea325a0-plugins-config-volume" pod "76edd686-a4f9-4da9-914d-01362ea325a0" (UID: "76edd686-a4f9-4da9-914d-01362ea325a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:49:05.589894 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.589866    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76edd686-a4f9-4da9-914d-01362ea325a0-kube-api-access-qzmqg" pod "76edd686-a4f9-4da9-914d-01362ea325a0" (UID: "76edd686-a4f9-4da9-914d-01362ea325a0"). InnerVolumeSpecName "kube-api-access-qzmqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:49:05.681744 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.681706    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0-kube-api-access-4pjcf\" (UniqueName: \"kubernetes.io/projected/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0-kube-api-access-4pjcf\") pod \"f06b87bd-ba8a-42cc-a4ef-0b6156b425a0\" (UID: \"f06b87bd-ba8a-42cc-a4ef-0b6156b425a0\") "
Oct 31 04:49:05.681982 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.681915    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0-plugins-config-volume\") pod \"f06b87bd-ba8a-42cc-a4ef-0b6156b425a0\" (UID: \"f06b87bd-ba8a-42cc-a4ef-0b6156b425a0\") "
Oct 31 04:49:05.682136 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.682119    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76edd686-a4f9-4da9-914d-01362ea325a0-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:05.682136 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.682135    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzmqg\" (UniqueName: \"kubernetes.io/projected/76edd686-a4f9-4da9-914d-01362ea325a0-kube-api-access-qzmqg\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:05.682283 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.682261    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0-plugins-config-volume" pod "f06b87bd-ba8a-42cc-a4ef-0b6156b425a0" (UID: "f06b87bd-ba8a-42cc-a4ef-0b6156b425a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:49:05.683903 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.683881    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0-kube-api-access-4pjcf" pod "f06b87bd-ba8a-42cc-a4ef-0b6156b425a0" (UID: "f06b87bd-ba8a-42cc-a4ef-0b6156b425a0"). InnerVolumeSpecName "kube-api-access-4pjcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:49:05.783301 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.783258    4900 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0-plugins-config-volume\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:05.783301 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:05.783293    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pjcf\" (UniqueName: \"kubernetes.io/projected/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0-kube-api-access-4pjcf\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:49:06.439096 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:06.439064    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"
Oct 31 04:49:06.439585 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:06.439090    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j" event={"ID":"76edd686-a4f9-4da9-914d-01362ea325a0","Type":"ContainerDied","Data":"c4e0aca464cc684056a7de77a6e36deb463b0cb73db8807b817e070fc58662df"}
Oct 31 04:49:06.439585 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:06.439174    4900 scope.go:122] "RemoveContainer" containerID="f5b243b0dc65c275163b2130f833a379f70ae955e20c0599cc492561652af296"
Oct 31 04:49:06.440640 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:06.440620    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb" event={"ID":"f06b87bd-ba8a-42cc-a4ef-0b6156b425a0","Type":"ContainerDied","Data":"f3a7d03ac26127deff707e87d788151308653ec30836f4eee928be92b22c245f"}
Oct 31 04:49:06.440753 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:06.440668    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"
Oct 31 04:49:06.449220 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:06.449201    4900 scope.go:122] "RemoveContainer" containerID="ba8e922a34da2ff224ad1215a847db9913204d5c90e6fc108789fa83f867d73c"
Oct 31 04:49:06.602421 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:06.602388    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"]
Oct 31 04:49:06.652475 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:06.652431    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-cc1991-69cc955kxb"]
Oct 31 04:49:06.762321 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:06.762283    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"]
Oct 31 04:49:06.765865 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:06.765840    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-cc1991-6f7b988744n88j"]
Oct 31 04:49:06.984218 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:49:06.984182    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:49:07.151951 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:07.151919    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76edd686-a4f9-4da9-914d-01362ea325a0" path="/var/lib/kubelet/pods/76edd686-a4f9-4da9-914d-01362ea325a0/volumes"
Oct 31 04:49:07.152363 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:07.152349    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06b87bd-ba8a-42cc-a4ef-0b6156b425a0" path="/var/lib/kubelet/pods/f06b87bd-ba8a-42cc-a4ef-0b6156b425a0/volumes"
Oct 31 04:49:16.151269 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:16.151231    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:16.373334 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:16.373295    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:16.930318 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:16.930276    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:17.254145 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:17.254109    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:17.618028 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:17.617943    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:18.809425 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:18.809391    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:19.157410 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:19.157322    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:19.759393 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:19.759349    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:22.332644 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:22.332608    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:22.550700 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:49:22.550665    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:49:37.025085 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:49:37.025043    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:50:06.983602 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:50:06.983497    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:50:17.284280 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:50:17.284245    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:17.495767 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:50:17.495726    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:50:36.984209 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:50:36.984168    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:51:07.025120 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:51:07.025082    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:51:12.842215 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:12.842131    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:13.034118 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:13.034083    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:17.623357 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:17.623315    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:51:24.150266 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:24.149885    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6rp2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:51:24.150266 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:24.149955    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d" podUID="10f0f3ad-038c-4104-ad7c-c0354f33862a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:51:36.984242 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:51:36.984203    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:51:43.811297 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.811262    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"]
Oct 31 04:51:43.811838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.811641    4900 cpu_manager.go:414] "containerMap: removing container" podUID="4b88d320-6ade-4021-8944-768c3e68c1b7" containerName="main"
Oct 31 04:51:43.811838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.811653    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b88d320-6ade-4021-8944-768c3e68c1b7" containerName="main"
Oct 31 04:51:43.811838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.811671    4900 cpu_manager.go:414] "containerMap: removing container" podUID="76edd686-a4f9-4da9-914d-01362ea325a0" containerName="payload-processing"
Oct 31 04:51:43.811838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.811676    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76edd686-a4f9-4da9-914d-01362ea325a0" containerName="payload-processing"
Oct 31 04:51:43.811838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.811689    4900 cpu_manager.go:414] "containerMap: removing container" podUID="34cc281c-75e7-43e4-a935-afbaa9be505d" containerName="maas-api"
Oct 31 04:51:43.811838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.811694    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34cc281c-75e7-43e4-a935-afbaa9be505d" containerName="maas-api"
Oct 31 04:51:43.811838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.811712    4900 cpu_manager.go:414] "containerMap: removing container" podUID="f06b87bd-ba8a-42cc-a4ef-0b6156b425a0" containerName="payload-pre-processing"
Oct 31 04:51:43.811838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.811716    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06b87bd-ba8a-42cc-a4ef-0b6156b425a0" containerName="payload-pre-processing"
Oct 31 04:51:43.811838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.811773    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="76edd686-a4f9-4da9-914d-01362ea325a0" containerName="payload-processing"
Oct 31 04:51:43.811838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.811781    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="34cc281c-75e7-43e4-a935-afbaa9be505d" containerName="maas-api"
Oct 31 04:51:43.811838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.811804    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06b87bd-ba8a-42cc-a4ef-0b6156b425a0" containerName="payload-pre-processing"
Oct 31 04:51:43.811838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.811813    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b88d320-6ade-4021-8944-768c3e68c1b7" containerName="main"
Oct 31 04:51:43.814940 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.814923    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:43.851849 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.851810    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f2eea5-0094-4336-abe5-c391694544b7-maas-api-tls\") pod \"maas-api-85868cf77f-fbmrl\" (UID: \"65f2eea5-0094-4336-abe5-c391694544b7\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:43.852054 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.851879    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trx6\" (UniqueName: \"kubernetes.io/projected/65f2eea5-0094-4336-abe5-c391694544b7-kube-api-access-9trx6\") pod \"maas-api-85868cf77f-fbmrl\" (UID: \"65f2eea5-0094-4336-abe5-c391694544b7\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:43.852054 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.851983    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f2eea5-0094-4336-abe5-c391694544b7-openshift-service-ca\") pod \"maas-api-85868cf77f-fbmrl\" (UID: \"65f2eea5-0094-4336-abe5-c391694544b7\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:43.852054 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.852021    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f2eea5-0094-4336-abe5-c391694544b7-metrics-certs\") pod \"maas-api-85868cf77f-fbmrl\" (UID: \"65f2eea5-0094-4336-abe5-c391694544b7\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:43.919411 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.919373    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"]
Oct 31 04:51:43.952909 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.952874    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f2eea5-0094-4336-abe5-c391694544b7-openshift-service-ca\") pod \"maas-api-85868cf77f-fbmrl\" (UID: \"65f2eea5-0094-4336-abe5-c391694544b7\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:43.953095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.952925    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f2eea5-0094-4336-abe5-c391694544b7-metrics-certs\") pod \"maas-api-85868cf77f-fbmrl\" (UID: \"65f2eea5-0094-4336-abe5-c391694544b7\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:43.953095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.952978    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f2eea5-0094-4336-abe5-c391694544b7-maas-api-tls\") pod \"maas-api-85868cf77f-fbmrl\" (UID: \"65f2eea5-0094-4336-abe5-c391694544b7\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:43.953095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.953010    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trx6\" (UniqueName: \"kubernetes.io/projected/65f2eea5-0094-4336-abe5-c391694544b7-kube-api-access-9trx6\") pod \"maas-api-85868cf77f-fbmrl\" (UID: \"65f2eea5-0094-4336-abe5-c391694544b7\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:43.953585 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.953550    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f2eea5-0094-4336-abe5-c391694544b7-openshift-service-ca\") pod \"maas-api-85868cf77f-fbmrl\" (UID: \"65f2eea5-0094-4336-abe5-c391694544b7\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:43.955508 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.955485    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65f2eea5-0094-4336-abe5-c391694544b7-metrics-certs\") pod \"maas-api-85868cf77f-fbmrl\" (UID: \"65f2eea5-0094-4336-abe5-c391694544b7\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:43.955606 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:43.955559    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65f2eea5-0094-4336-abe5-c391694544b7-maas-api-tls\") pod \"maas-api-85868cf77f-fbmrl\" (UID: \"65f2eea5-0094-4336-abe5-c391694544b7\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:44.009483 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:44.009453    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trx6\" (UniqueName: \"kubernetes.io/projected/65f2eea5-0094-4336-abe5-c391694544b7-kube-api-access-9trx6\") pod \"maas-api-85868cf77f-fbmrl\" (UID: \"65f2eea5-0094-4336-abe5-c391694544b7\") " pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:44.127355 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:44.127263    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:44.495554 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:51:44.495510    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f2eea5_0094_4336_abe5_c391694544b7.slice/crio-bd4acc4200ba6df3dee495dfc271875e86aba40325a7af579d645c51716a1a7f WatchSource:0}: Error finding container bd4acc4200ba6df3dee495dfc271875e86aba40325a7af579d645c51716a1a7f: Status 404 returned error can't find the container with id bd4acc4200ba6df3dee495dfc271875e86aba40325a7af579d645c51716a1a7f
Oct 31 04:51:44.497542 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:44.497518    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"]
Oct 31 04:51:45.225616 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:45.225578    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl" event={"ID":"65f2eea5-0094-4336-abe5-c391694544b7","Type":"ContainerStarted","Data":"bd4acc4200ba6df3dee495dfc271875e86aba40325a7af579d645c51716a1a7f"}
Oct 31 04:51:46.231435 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:46.231399    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl" event={"ID":"65f2eea5-0094-4336-abe5-c391694544b7","Type":"ContainerStarted","Data":"7528db1f8dd048bd5e092072ac4739b42bdc0ecc991c79acc9f79b5877432da6"}
Oct 31 04:51:46.231896 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:46.231591    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:46.358680 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:46.358624    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl" podStartSLOduration=2.41824703 podStartE2EDuration="3.35860689s" totalImagesPullingTime="940.35986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:51:46.357768701 +0000 UTC m=+6259.324102046" watchObservedRunningTime="2026-10-31 04:51:46.35860689 +0000 UTC m=+6259.324940233"
Oct 31 04:51:52.241357 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:52.241327    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-85868cf77f-fbmrl"
Oct 31 04:51:52.407126 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:52.407093    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-997c84955-lk8d7"]
Oct 31 04:51:52.407318 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:52.407296    4900 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7" podUID="9f14a322-a62a-464a-bbb5-5c9e60320977" containerName="maas-api" containerID="cri-o://2ecb6e8321237693d2e789b3a109a80635613a862a6d1316a23cb53e43cb6750" gracePeriod=30
Oct 31 04:51:52.634146 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:52.634061    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7" podUID="9f14a322-a62a-464a-bbb5-5c9e60320977" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.22:8443/health\": dial tcp 10.133.0.22:8443: connect: connection refused"
Oct 31 04:51:53.660384 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:53.660361    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 04:51:53.747612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:53.747528    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f14a322-a62a-464a-bbb5-5c9e60320977-kube-api-access-svm9q\" (UniqueName: \"kubernetes.io/projected/9f14a322-a62a-464a-bbb5-5c9e60320977-kube-api-access-svm9q\") pod \"9f14a322-a62a-464a-bbb5-5c9e60320977\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") "
Oct 31 04:51:53.747612 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:53.747585    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f14a322-a62a-464a-bbb5-5c9e60320977-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f14a322-a62a-464a-bbb5-5c9e60320977-openshift-service-ca\") pod \"9f14a322-a62a-464a-bbb5-5c9e60320977\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") "
Oct 31 04:51:53.747867 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:53.747665    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f14a322-a62a-464a-bbb5-5c9e60320977-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f14a322-a62a-464a-bbb5-5c9e60320977-metrics-certs\") pod \"9f14a322-a62a-464a-bbb5-5c9e60320977\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") "
Oct 31 04:51:53.747867 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:53.747728    4900 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f14a322-a62a-464a-bbb5-5c9e60320977-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f14a322-a62a-464a-bbb5-5c9e60320977-maas-api-tls\") pod \"9f14a322-a62a-464a-bbb5-5c9e60320977\" (UID: \"9f14a322-a62a-464a-bbb5-5c9e60320977\") "
Oct 31 04:51:53.747997 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:53.747965    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f14a322-a62a-464a-bbb5-5c9e60320977-openshift-service-ca" pod "9f14a322-a62a-464a-bbb5-5c9e60320977" (UID: "9f14a322-a62a-464a-bbb5-5c9e60320977"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:51:53.748180 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:53.748160    4900 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f14a322-a62a-464a-bbb5-5c9e60320977-openshift-service-ca\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:51:53.749817 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:53.749765    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f14a322-a62a-464a-bbb5-5c9e60320977-maas-api-tls" pod "9f14a322-a62a-464a-bbb5-5c9e60320977" (UID: "9f14a322-a62a-464a-bbb5-5c9e60320977"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:51:53.749817 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:53.749802    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f14a322-a62a-464a-bbb5-5c9e60320977-metrics-certs" pod "9f14a322-a62a-464a-bbb5-5c9e60320977" (UID: "9f14a322-a62a-464a-bbb5-5c9e60320977"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:51:53.749968 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:53.749846    4900 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f14a322-a62a-464a-bbb5-5c9e60320977-kube-api-access-svm9q" pod "9f14a322-a62a-464a-bbb5-5c9e60320977" (UID: "9f14a322-a62a-464a-bbb5-5c9e60320977"). InnerVolumeSpecName "kube-api-access-svm9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:51:53.848750 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:53.848722    4900 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svm9q\" (UniqueName: \"kubernetes.io/projected/9f14a322-a62a-464a-bbb5-5c9e60320977-kube-api-access-svm9q\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:51:53.848750 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:53.848753    4900 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f14a322-a62a-464a-bbb5-5c9e60320977-metrics-certs\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:51:53.848969 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:53.848766    4900 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f14a322-a62a-464a-bbb5-5c9e60320977-maas-api-tls\") on node \"ip-10-0-134-83.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 04:51:54.269558 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:54.269528    4900 generic.go:356] "Generic (PLEG): container finished" podID="9f14a322-a62a-464a-bbb5-5c9e60320977" containerID="2ecb6e8321237693d2e789b3a109a80635613a862a6d1316a23cb53e43cb6750" exitCode=0
Oct 31 04:51:54.269764 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:54.269574    4900 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7"
Oct 31 04:51:54.269764 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:54.269618    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7" event={"ID":"9f14a322-a62a-464a-bbb5-5c9e60320977","Type":"ContainerDied","Data":"2ecb6e8321237693d2e789b3a109a80635613a862a6d1316a23cb53e43cb6750"}
Oct 31 04:51:54.269764 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:54.269675    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-997c84955-lk8d7" event={"ID":"9f14a322-a62a-464a-bbb5-5c9e60320977","Type":"ContainerDied","Data":"9773e6149e953a0ad606e9120f1d9e89388a0f0c24dffdc94623c8f20516f82a"}
Oct 31 04:51:54.269764 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:54.269703    4900 scope.go:122] "RemoveContainer" containerID="2ecb6e8321237693d2e789b3a109a80635613a862a6d1316a23cb53e43cb6750"
Oct 31 04:51:54.281020 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:54.280995    4900 scope.go:122] "RemoveContainer" containerID="2ecb6e8321237693d2e789b3a109a80635613a862a6d1316a23cb53e43cb6750"
Oct 31 04:51:54.281310 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:51:54.281286    4900 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ecb6e8321237693d2e789b3a109a80635613a862a6d1316a23cb53e43cb6750\": container with ID starting with 2ecb6e8321237693d2e789b3a109a80635613a862a6d1316a23cb53e43cb6750 not found: ID does not exist" containerID="2ecb6e8321237693d2e789b3a109a80635613a862a6d1316a23cb53e43cb6750"
Oct 31 04:51:54.281365 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:54.281317    4900 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ecb6e8321237693d2e789b3a109a80635613a862a6d1316a23cb53e43cb6750"} err="failed to get container status \"2ecb6e8321237693d2e789b3a109a80635613a862a6d1316a23cb53e43cb6750\": rpc error: code = NotFound desc = could not find container \"2ecb6e8321237693d2e789b3a109a80635613a862a6d1316a23cb53e43cb6750\": container with ID starting with 2ecb6e8321237693d2e789b3a109a80635613a862a6d1316a23cb53e43cb6750 not found: ID does not exist"
Oct 31 04:51:54.431999 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:54.431971    4900 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-997c84955-lk8d7"]
Oct 31 04:51:54.434438 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:54.434415    4900 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-997c84955-lk8d7"]
Oct 31 04:51:55.158044 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:51:55.153227    4900 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f14a322-a62a-464a-bbb5-5c9e60320977" path="/var/lib/kubelet/pods/9f14a322-a62a-464a-bbb5-5c9e60320977/volumes"
Oct 31 04:52:02.602948 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:52:02.602907    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-9sp48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:52:02.603565 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:52:02.603526    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-9sp48" podUID="ae6469aa-5349-41a7-9cb8-3ec92f92043c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:52:06.983966 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:52:06.983927    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:52:17.972665 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:52:17.972601    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:22.672546 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:52:22.672515    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:22.872490 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:52:22.872460    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:52:29.138122 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:52:29.138090    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:52:29.143367 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:52:29.143342    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:52:36.983993 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:52:36.983954    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:53:06.985191 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:53:06.985151    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:53:18.383529 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:53:18.383480    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:53:25.611015 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:53:25.610964    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lxq9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:53:25.611562 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:53:25.611055    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" podUID="d7f8ce75-4372-4c11-aea6-6922ed91ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:53:36.983127 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:53:36.983089    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:54:06.984464 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:54:06.984428    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:54:10.611138 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:54:10.610930    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lxq9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:54:10.611869 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:54:10.611833    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" podUID="d7f8ce75-4372-4c11-aea6-6922ed91ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:54:18.741990 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:54:18.741948    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:54:24.138438 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:54:24.137946    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6rp2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:54:24.138438 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:54:24.138012    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d" podUID="10f0f3ad-038c-4104-ad7c-c0354f33862a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:54:36.983899 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:54:36.983860    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:54:56.857230 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:54:56.857195    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:06.983278 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:55:06.983240    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:55:09.151312 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:55:09.151267    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-6rp2d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:55:09.153855 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:55:09.151850    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-6rp2d" podUID="10f0f3ad-038c-4104-ad7c-c0354f33862a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:55:19.155952 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:55:19.155918    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:19.365565 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:55:19.365533    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:55:36.983757 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:55:36.983712    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:56:06.983112 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:56:06.983071    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:56:09.911707 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:56:09.911673    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:15.331065 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:56:15.331028    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:15.540811 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:56:15.540747    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:19.497450 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:56:19.497412    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:56:36.983630 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:56:36.983586    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:57:06.983245 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:57:06.983205    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:57:19.850256 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:57:19.850171    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:57:25.613016 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:57:25.612971    4900 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6886b96f9-lxq9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:57:25.613654 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:57:25.613059    4900 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6886b96f9-lxq9w" podUID="d7f8ce75-4372-4c11-aea6-6922ed91ba8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:57:29.173693 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:57:29.173652    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:57:29.180756 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:57:29.180727    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xm5zs_cda5dcc5-e5c6-4e0a-9acb-6191475f552c/ovn-acl-logging/0.log"
Oct 31 04:57:36.982853 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:57:36.982815    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:58:06.983406 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:58:06.983369    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:58:20.262160 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:20.262120    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:21.538838 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:21.538783    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:26.824599 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:26.824559    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5cb476b87b-tk2rk_b7b8d483-9270-45ec-94dc-d3e1bc625ca2/manager/0.log"
Oct 31 04:58:29.956937 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:29.956906    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-85868cf77f-fbmrl_65f2eea5-0094-4336-abe5-c391694544b7/maas-api/0.log"
Oct 31 04:58:36.983691 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:58:36.983647    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:58:39.065732 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:39.065642    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cb6b5fd89-vj58j_b158a0dc-55bd-4d81-bf1c-f83d4896dba5/kube-auth-proxy/0.log"
Oct 31 04:58:39.733766 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:39.733739    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6rp2d_10f0f3ad-038c-4104-ad7c-c0354f33862a/istio-proxy/0.log"
Oct 31 04:58:39.968468 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:39.968431    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-9sp48_ae6469aa-5349-41a7-9cb8-3ec92f92043c/istio-proxy/0.log"
Oct 31 04:58:41.121691 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:41.121660    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-lxq9w_d7f8ce75-4372-4c11-aea6-6922ed91ba8a/istio-proxy/0.log"
Oct 31 04:58:42.033097 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:42.033063    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-9cfb9b656-jtls9_d926de8d-6439-41a4-8ad6-1c01833ed69c/payload-pre-processing/0.log"
Oct 31 04:58:42.256382 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:42.256353    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-79468f7c9f-559sl_f54b64bf-7962-4b4c-ab0f-d3c35d5142ed/payload-processing/0.log"
Oct 31 04:58:43.511501 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:43.511473    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-br9sv_42af8fc7-c7bd-4af5-b122-5dec2cd2dd85/main/0.log"
Oct 31 04:58:43.735182 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:43.735153    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-n42bp_520f2099-f186-4222-a31a-9f073c33dced/main/0.log"
Oct 31 04:58:43.951290 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:43.951215    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-psl4l_3792ff36-75f6-4fb5-b67b-ee4f086c6ef3/main/0.log"
Oct 31 04:58:44.166897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:44.166867    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-4mdxf_2f4f0318-9063-4c10-8c6a-240ed741b154/main/0.log"
Oct 31 04:58:44.376851 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:44.376778    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8v2xw5_37936d51-1fa0-49f0-9d91-f4f3d0e4a0e8/main/0.log"
Oct 31 04:58:44.593799 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:44.593757    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-ns4fr_246cfb6b-4be8-4f89-bcc4-58f24acf2229/main/0.log"
Oct 31 04:58:45.013078 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:45.013049    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-bvj7j_43738834-604f-4063-8c19-d210e1a44de7/main/0.log"
Oct 31 04:58:55.338074 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.338040    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6hhnf/must-gather-8cb6k"]
Oct 31 04:58:55.338480 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.338434    4900 cpu_manager.go:414] "containerMap: removing container" podUID="9f14a322-a62a-464a-bbb5-5c9e60320977" containerName="maas-api"
Oct 31 04:58:55.338480 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.338444    4900 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f14a322-a62a-464a-bbb5-5c9e60320977" containerName="maas-api"
Oct 31 04:58:55.338556 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.338519    4900 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f14a322-a62a-464a-bbb5-5c9e60320977" containerName="maas-api"
Oct 31 04:58:55.341744 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.341726    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6hhnf/must-gather-8cb6k"
Oct 31 04:58:55.417092 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.417063    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:55.444220 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.444190    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6hhnf\"/\"openshift-service-ca.crt\""
Oct 31 04:58:55.444220 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.444220    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6hhnf\"/\"default-dockercfg-skcpn\""
Oct 31 04:58:55.444415 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.444247    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6hhnf\"/\"kube-root-ca.crt\""
Oct 31 04:58:55.445889 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.445872    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6hhnf/must-gather-8cb6k"]
Oct 31 04:58:55.447193 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.447172    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8dgk4/must-gather-v7sh7"]
Oct 31 04:58:55.450658 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.450644    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8dgk4/must-gather-v7sh7"
Oct 31 04:58:55.474485 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.474454    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709-must-gather-output\") pod \"must-gather-8cb6k\" (UID: \"e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709\") " pod="openshift-must-gather-6hhnf/must-gather-8cb6k"
Oct 31 04:58:55.474624 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.474488    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsjkw\" (UniqueName: \"kubernetes.io/projected/e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709-kube-api-access-gsjkw\") pod \"must-gather-8cb6k\" (UID: \"e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709\") " pod="openshift-must-gather-6hhnf/must-gather-8cb6k"
Oct 31 04:58:55.495690 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.495666    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8dgk4/must-gather-v7sh7"]
Oct 31 04:58:55.576095 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.576054    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c5f90e49-65e9-43f4-9227-17e7e3b2efbd-must-gather-output\") pod \"must-gather-v7sh7\" (UID: \"c5f90e49-65e9-43f4-9227-17e7e3b2efbd\") " pod="openshift-must-gather-8dgk4/must-gather-v7sh7"
Oct 31 04:58:55.576277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.576112    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709-must-gather-output\") pod \"must-gather-8cb6k\" (UID: \"e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709\") " pod="openshift-must-gather-6hhnf/must-gather-8cb6k"
Oct 31 04:58:55.576277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.576168    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsjkw\" (UniqueName: \"kubernetes.io/projected/e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709-kube-api-access-gsjkw\") pod \"must-gather-8cb6k\" (UID: \"e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709\") " pod="openshift-must-gather-6hhnf/must-gather-8cb6k"
Oct 31 04:58:55.576406 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.576308    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbdwj\" (UniqueName: \"kubernetes.io/projected/c5f90e49-65e9-43f4-9227-17e7e3b2efbd-kube-api-access-kbdwj\") pod \"must-gather-v7sh7\" (UID: \"c5f90e49-65e9-43f4-9227-17e7e3b2efbd\") " pod="openshift-must-gather-8dgk4/must-gather-v7sh7"
Oct 31 04:58:55.576477 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.576460    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709-must-gather-output\") pod \"must-gather-8cb6k\" (UID: \"e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709\") " pod="openshift-must-gather-6hhnf/must-gather-8cb6k"
Oct 31 04:58:55.596552 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.596489    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8dgk4\"/\"kube-root-ca.crt\""
Oct 31 04:58:55.596552 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.596498    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8dgk4\"/\"default-dockercfg-jm7b6\""
Oct 31 04:58:55.596552 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.596494    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8dgk4\"/\"openshift-service-ca.crt\""
Oct 31 04:58:55.629602 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.629581    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:58:55.632544 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.632525    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsjkw\" (UniqueName: \"kubernetes.io/projected/e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709-kube-api-access-gsjkw\") pod \"must-gather-8cb6k\" (UID: \"e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709\") " pod="openshift-must-gather-6hhnf/must-gather-8cb6k"
Oct 31 04:58:55.655371 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.655350    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6hhnf/must-gather-8cb6k"
Oct 31 04:58:55.677107 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.677082    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c5f90e49-65e9-43f4-9227-17e7e3b2efbd-must-gather-output\") pod \"must-gather-v7sh7\" (UID: \"c5f90e49-65e9-43f4-9227-17e7e3b2efbd\") " pod="openshift-must-gather-8dgk4/must-gather-v7sh7"
Oct 31 04:58:55.677266 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.677247    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbdwj\" (UniqueName: \"kubernetes.io/projected/c5f90e49-65e9-43f4-9227-17e7e3b2efbd-kube-api-access-kbdwj\") pod \"must-gather-v7sh7\" (UID: \"c5f90e49-65e9-43f4-9227-17e7e3b2efbd\") " pod="openshift-must-gather-8dgk4/must-gather-v7sh7"
Oct 31 04:58:55.677490 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.677467    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c5f90e49-65e9-43f4-9227-17e7e3b2efbd-must-gather-output\") pod \"must-gather-v7sh7\" (UID: \"c5f90e49-65e9-43f4-9227-17e7e3b2efbd\") " pod="openshift-must-gather-8dgk4/must-gather-v7sh7"
Oct 31 04:58:55.734463 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.734392    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbdwj\" (UniqueName: \"kubernetes.io/projected/c5f90e49-65e9-43f4-9227-17e7e3b2efbd-kube-api-access-kbdwj\") pod \"must-gather-v7sh7\" (UID: \"c5f90e49-65e9-43f4-9227-17e7e3b2efbd\") " pod="openshift-must-gather-8dgk4/must-gather-v7sh7"
Oct 31 04:58:55.764607 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:55.764581    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8dgk4/must-gather-v7sh7"
Oct 31 04:58:56.021437 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:58:56.021412    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4b48bf9_fdfb_4d9d_a75c_9b7f312b7709.slice/crio-a9a5c793f0a4df242ea0f1a14766c951c3a883d8167deaccb3d81a6f42e7c860 WatchSource:0}: Error finding container a9a5c793f0a4df242ea0f1a14766c951c3a883d8167deaccb3d81a6f42e7c860: Status 404 returned error can't find the container with id a9a5c793f0a4df242ea0f1a14766c951c3a883d8167deaccb3d81a6f42e7c860
Oct 31 04:58:56.022774 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:56.022752    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6hhnf/must-gather-8cb6k"]
Oct 31 04:58:56.023544 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:56.023526    4900 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:58:56.128151 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:58:56.128124    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5f90e49_65e9_43f4_9227_17e7e3b2efbd.slice/crio-c3c4a10ed7ee09854f20ab06adf2f5fdb41846453869720019fb95d8d9a39540 WatchSource:0}: Error finding container c3c4a10ed7ee09854f20ab06adf2f5fdb41846453869720019fb95d8d9a39540: Status 404 returned error can't find the container with id c3c4a10ed7ee09854f20ab06adf2f5fdb41846453869720019fb95d8d9a39540
Oct 31 04:58:56.128510 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:56.128489    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8dgk4/must-gather-v7sh7"]
Oct 31 04:58:56.645874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:56.645837    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6hhnf/must-gather-8cb6k" event={"ID":"e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709","Type":"ContainerStarted","Data":"a9a5c793f0a4df242ea0f1a14766c951c3a883d8167deaccb3d81a6f42e7c860"}
Oct 31 04:58:56.647068 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:58:56.647043    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8dgk4/must-gather-v7sh7" event={"ID":"c5f90e49-65e9-43f4-9227-17e7e3b2efbd","Type":"ContainerStarted","Data":"c3c4a10ed7ee09854f20ab06adf2f5fdb41846453869720019fb95d8d9a39540"}
Oct 31 04:59:04.706298 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:04.706255    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8dgk4/must-gather-v7sh7" event={"ID":"c5f90e49-65e9-43f4-9227-17e7e3b2efbd","Type":"ContainerStarted","Data":"b96bf5ecccc2d37b920786ee64ba4dafd9de71ead7b87fda65466436fa55b6d5"}
Oct 31 04:59:04.706865 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:04.706318    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8dgk4/must-gather-v7sh7" event={"ID":"c5f90e49-65e9-43f4-9227-17e7e3b2efbd","Type":"ContainerStarted","Data":"086a05ada1b77751e9a22fe6e1313edf40d5ae88f4e695697ac4fb4e7ffbe813"}
Oct 31 04:59:04.708048 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:04.708020    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6hhnf/must-gather-8cb6k" event={"ID":"e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709","Type":"ContainerStarted","Data":"cd5f9788816479777f0f5d07c11d3b1921dedafd6ac1f5a2ad8df06089248ae5"}
Oct 31 04:59:04.708161 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:04.708061    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6hhnf/must-gather-8cb6k" event={"ID":"e4b48bf9-fdfb-4d9d-a75c-9b7f312b7709","Type":"ContainerStarted","Data":"f07f322bba9be05282eb3edaa235007aa8104bcc006cf2c26ad7f9e6363035dc"}
Oct 31 04:59:04.819563 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:04.819507    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8dgk4/must-gather-v7sh7" podStartSLOduration=2.161752502 podStartE2EDuration="9.819493583s" totalImagesPullingTime="7.657741081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:59:04.818519952 +0000 UTC m=+6697.784853296" watchObservedRunningTime="2026-10-31 04:59:04.819493583 +0000 UTC m=+6697.785826923"
Oct 31 04:59:04.930397 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:04.930345    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-6hhnf/must-gather-8cb6k" podStartSLOduration=2.103996921 podStartE2EDuration="9.930331853s" totalImagesPullingTime="7.826334932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:58:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:59:04.929755877 +0000 UTC m=+6697.896089222" watchObservedRunningTime="2026-10-31 04:59:04.930331853 +0000 UTC m=+6697.896665196"
Oct 31 04:59:05.086144 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:05.086110    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ndfgt_f714c9b3-ccf2-4dab-9220-321c5b4258d0/global-pull-secret-syncer/0.log"
Oct 31 04:59:06.941377 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:06.941343    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log"
Oct 31 04:59:06.989191 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:59:06.989155    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:59:07.728108 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:07.728076    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5cb476b87b-tk2rk_b7b8d483-9270-45ec-94dc-d3e1bc625ca2/manager/0.log"
Oct 31 04:59:07.851469 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:07.851440    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5cb476b87b-tk2rk_b7b8d483-9270-45ec-94dc-d3e1bc625ca2/manager/0.log"
Oct 31 04:59:08.128288 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:08.128256    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-83.us-west-2.compute.internal_584715508ebcb798df8a2f82469ae3b6/haproxy/0.log"
Oct 31 04:59:15.176146 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.176114    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5cb476b87b-tk2rk_b7b8d483-9270-45ec-94dc-d3e1bc625ca2/manager/0.log"
Oct 31 04:59:15.184414 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.180405    4900 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"]
Oct 31 04:59:15.189499 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.189471    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.241808 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.241752    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m7rwt\"/\"openshift-service-ca.crt\""
Oct 31 04:59:15.290557 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.290517    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77d720e8-6d12-4d14-b5fd-241a2e585496-proc\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.290750 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.290574    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77d720e8-6d12-4d14-b5fd-241a2e585496-sys\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.290874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.290746    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65xzn\" (UniqueName: \"kubernetes.io/projected/77d720e8-6d12-4d14-b5fd-241a2e585496-kube-api-access-65xzn\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.290874 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.290799    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77d720e8-6d12-4d14-b5fd-241a2e585496-lib-modules\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.291006 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.290981    4900 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77d720e8-6d12-4d14-b5fd-241a2e585496-podres\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.291895 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.291867    4900 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m7rwt\"/\"kube-root-ca.crt\""
Oct 31 04:59:15.292048 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.291902    4900 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m7rwt\"/\"default-dockercfg-74bbv\""
Oct 31 04:59:15.293523 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.293501    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"]
Oct 31 04:59:15.369267 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.369242    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5cb476b87b-tk2rk_b7b8d483-9270-45ec-94dc-d3e1bc625ca2/manager/0.log"
Oct 31 04:59:15.391725 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.391702    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77d720e8-6d12-4d14-b5fd-241a2e585496-proc\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.391870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.391735    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77d720e8-6d12-4d14-b5fd-241a2e585496-sys\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.391870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.391767    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65xzn\" (UniqueName: \"kubernetes.io/projected/77d720e8-6d12-4d14-b5fd-241a2e585496-kube-api-access-65xzn\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.391870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.391839    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77d720e8-6d12-4d14-b5fd-241a2e585496-proc\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.391870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.391846    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77d720e8-6d12-4d14-b5fd-241a2e585496-sys\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.391870 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.391869    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77d720e8-6d12-4d14-b5fd-241a2e585496-lib-modules\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.392106 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.391909    4900 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77d720e8-6d12-4d14-b5fd-241a2e585496-podres\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.392106 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.392012    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77d720e8-6d12-4d14-b5fd-241a2e585496-lib-modules\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.392106 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.392024    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77d720e8-6d12-4d14-b5fd-241a2e585496-podres\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.449330 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.449267    4900 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65xzn\" (UniqueName: \"kubernetes.io/projected/77d720e8-6d12-4d14-b5fd-241a2e585496-kube-api-access-65xzn\") pod \"perf-node-gather-daemonset-pzjhz\" (UID: \"77d720e8-6d12-4d14-b5fd-241a2e585496\") " pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.501194 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.501168    4900 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:15.867701 ip-10-0-134-83 kubenswrapper[4900]: W1031 04:59:15.867672    4900 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod77d720e8_6d12_4d14_b5fd_241a2e585496.slice/crio-287f76f87088db2abfee9d30627c449beca45c454049b4c69c7c46d97f825f84 WatchSource:0}: Error finding container 287f76f87088db2abfee9d30627c449beca45c454049b4c69c7c46d97f825f84: Status 404 returned error can't find the container with id 287f76f87088db2abfee9d30627c449beca45c454049b4c69c7c46d97f825f84
Oct 31 04:59:15.870079 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:15.870054    4900 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"]
Oct 31 04:59:16.769295 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:16.769261    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz" event={"ID":"77d720e8-6d12-4d14-b5fd-241a2e585496","Type":"ContainerStarted","Data":"8d0c853990e1f6709b4e945e0742a118ab517d8ae659e078242c2abaeb3c1fb9"}
Oct 31 04:59:16.769684 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:16.769310    4900 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz" event={"ID":"77d720e8-6d12-4d14-b5fd-241a2e585496","Type":"ContainerStarted","Data":"287f76f87088db2abfee9d30627c449beca45c454049b4c69c7c46d97f825f84"}
Oct 31 04:59:16.769684 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:16.769395    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:16.885151 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:16.885098    4900 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz" podStartSLOduration=1.885082863 podStartE2EDuration="1.885082863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:59:16.88361399 +0000 UTC m=+6709.849947334" watchObservedRunningTime="2026-10-31 04:59:16.885082863 +0000 UTC m=+6709.851416207"
Oct 31 04:59:18.878720 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:18.878687    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_16dab515-6397-4259-bfff-7cb9e0bd83de/alertmanager/0.log"
Oct 31 04:59:19.285835 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:19.284496    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_16dab515-6397-4259-bfff-7cb9e0bd83de/config-reloader/0.log"
Oct 31 04:59:19.697025 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:19.696944    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_16dab515-6397-4259-bfff-7cb9e0bd83de/kube-rbac-proxy-web/0.log"
Oct 31 04:59:20.102850 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:20.102816    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_16dab515-6397-4259-bfff-7cb9e0bd83de/kube-rbac-proxy/0.log"
Oct 31 04:59:20.507756 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:20.507728    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_16dab515-6397-4259-bfff-7cb9e0bd83de/kube-rbac-proxy-metric/0.log"
Oct 31 04:59:20.567341 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:20.567314    4900 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c7b69_f5c68fab-89c6-485c-a250-cb958115e5a2/konnectivity-agent/0.log" limit=1024
Oct 31 04:59:20.912277 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:20.912207    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_16dab515-6397-4259-bfff-7cb9e0bd83de/prom-label-proxy/0.log"
Oct 31 04:59:21.317356 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:21.317323    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_16dab515-6397-4259-bfff-7cb9e0bd83de/init-config-reloader/0.log"
Oct 31 04:59:21.998660 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:21.998628    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5cb476b87b-tk2rk_b7b8d483-9270-45ec-94dc-d3e1bc625ca2/manager/0.log"
Oct 31 04:59:22.103995 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:22.103968    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-5cb476b87b-tk2rk_b7b8d483-9270-45ec-94dc-d3e1bc625ca2/manager/0.log"
Oct 31 04:59:22.784772 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:22.784744    4900 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m7rwt/perf-node-gather-daemonset-pzjhz"
Oct 31 04:59:26.598093 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:26.598066    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wwhp2_577fcc94-dce8-4841-bf6a-7102e073b362/node-exporter/0.log"
Oct 31 04:59:27.002171 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:27.002142    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wwhp2_577fcc94-dce8-4841-bf6a-7102e073b362/kube-rbac-proxy/0.log"
Oct 31 04:59:27.406897 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:27.406824    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wwhp2_577fcc94-dce8-4841-bf6a-7102e073b362/init-textfile/0.log"
Oct 31 04:59:33.093204 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:33.093169    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f9c76d5c-7fddq_7789c29a-d4b9-41b9-8a8a-6e48255f6e7d/telemeter-client/0.log"
Oct 31 04:59:33.499051 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:33.499020    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f9c76d5c-7fddq_7789c29a-d4b9-41b9-8a8a-6e48255f6e7d/reload/0.log"
Oct 31 04:59:33.906962 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:33.906885    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f9c76d5c-7fddq_7789c29a-d4b9-41b9-8a8a-6e48255f6e7d/kube-rbac-proxy/0.log"
Oct 31 04:59:34.261173 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:34.261146    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cb6b5fd89-vj58j_b158a0dc-55bd-4d81-bf1c-f83d4896dba5/kube-auth-proxy/0.log"
Oct 31 04:59:34.321594 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:34.321560    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cd5fd947f-966d4_b29ff389-bc68-4df1-86e3-0ea1c48b4182/thanos-query/0.log"
Oct 31 04:59:34.573352 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:34.573258    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cb6b5fd89-vj58j_b158a0dc-55bd-4d81-bf1c-f83d4896dba5/kube-auth-proxy/0.log"
Oct 31 04:59:34.727231 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:34.727190    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cd5fd947f-966d4_b29ff389-bc68-4df1-86e3-0ea1c48b4182/kube-rbac-proxy-web/0.log"
Oct 31 04:59:34.887005 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:34.886933    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6rp2d_10f0f3ad-038c-4104-ad7c-c0354f33862a/istio-proxy/0.log"
Oct 31 04:59:35.095286 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:35.095256    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-9sp48_ae6469aa-5349-41a7-9cb8-3ec92f92043c/istio-proxy/0.log"
Oct 31 04:59:35.132636 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:35.132609    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cd5fd947f-966d4_b29ff389-bc68-4df1-86e3-0ea1c48b4182/kube-rbac-proxy/0.log"
Oct 31 04:59:35.194384 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:35.194302    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-6rp2d_10f0f3ad-038c-4104-ad7c-c0354f33862a/istio-proxy/0.log"
Oct 31 04:59:35.400892 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:35.400867    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-9sp48_ae6469aa-5349-41a7-9cb8-3ec92f92043c/istio-proxy/0.log"
Oct 31 04:59:35.536851 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:35.536820    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cd5fd947f-966d4_b29ff389-bc68-4df1-86e3-0ea1c48b4182/prom-label-proxy/0.log"
Oct 31 04:59:35.942209 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:35.942127    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cd5fd947f-966d4_b29ff389-bc68-4df1-86e3-0ea1c48b4182/kube-rbac-proxy-rules/0.log"
Oct 31 04:59:36.137292 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:36.137264    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-lxq9w_d7f8ce75-4372-4c11-aea6-6922ed91ba8a/istio-proxy/0.log"
Oct 31 04:59:36.348090 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:36.348063    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5cd5fd947f-966d4_b29ff389-bc68-4df1-86e3-0ea1c48b4182/kube-rbac-proxy-metrics/0.log"
Oct 31 04:59:36.443539 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:36.443511    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6886b96f9-lxq9w_d7f8ce75-4372-4c11-aea6-6922ed91ba8a/istio-proxy/0.log"
Oct 31 04:59:36.961833 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:36.961802    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-9cfb9b656-jtls9_d926de8d-6439-41a4-8ad6-1c01833ed69c/payload-pre-processing/0.log"
Oct 31 04:59:36.984149 ip-10-0-134-83 kubenswrapper[4900]: E1031 04:59:36.984117    4900 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-8z98n" containerName="extractor"
Oct 31 04:59:37.168398 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:37.168367    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-79468f7c9f-559sl_f54b64bf-7962-4b4c-ab0f-d3c35d5142ed/payload-processing/0.log"
Oct 31 04:59:37.271334 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:37.271307    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-9cfb9b656-jtls9_d926de8d-6439-41a4-8ad6-1c01833ed69c/payload-pre-processing/0.log"
Oct 31 04:59:37.483275 ip-10-0-134-83 kubenswrapper[4900]: I1031 04:59:37.483249    4900 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-79468f7c9f-559sl_f54b64bf-7962-4b4c-ab0f-d3c35d5142ed/payload-processing/0.log"